development.rst

Latest commit

History

History
224 lines (162 loc) · 8.44 KB

development.rst

File metadata and controls

224 lines (162 loc) · 8.44 KB
Raw

Development Guidelines

This section provides information for developers who wish to contribute to the C++ codebase.

Note

Since most of the project's developers work on Linux or macOS, not all features or developer tools are uniformly supported on Windows. If you are on Windows, have a look at :ref:`developers-cpp-windows`.

Compiler warning levels

The BUILD_WARNING_LEVEL CMake option switches between sets of predetermined compiler warning levels that we use for code tidiness. For release builds, the default warning level is PRODUCTION, while for debug builds the default is CHECKIN.

When using CHECKIN for debug builds, -Werror is added when using gcc and clang, causing build failures for any warning, and /WX is set with MSVC having the same effect.

Running unit tests

The -DARROW_BUILD_TESTS=ON CMake option enables building of unit test executables. You can then either run them individually, by launching the desired executable, or run them all at once by launching the ctest executable (which is part of the CMake suite).

A possible invocation is something like:

$ ctest -j16 --output-on-failure

where the -j16 option runs up to 16 tests in parallel, taking advantage of multiple CPU cores and hardware threads.

Running benchmarks

The -DARROW_BUILD_BENCHMARKS=ON CMake option enables building of benchmark executables. You can then run benchmarks individually by launching the corresponding executable from the command line, e.g.:

$ ./build/release/arrow-builder-benchmark

Note

For meaningful benchmark numbers, it is very strongly recommended to build in Release mode, so as to enable compiler optimizations.

Code Style, Linting, and CI

This project follows Google's C++ Style Guide with these exceptions:

  • We relax the line length restriction to 90 characters.
  • We use the NULLPTR macro in header files (instead of nullptr) defined in src/arrow/util/macros.h to support building C++/CLI (ARROW-1134).
  • We relax the guide's rules regarding structs. For public headers we should use struct only for objects that are principally simple data containers where it is OK to expose all the internal members and any methods are primarily conveniences. For private headers the rules are relaxed further and structs can be used where convenient for types that do not need access control even though they may not be simple data containers.
  • We prefer pointers for output and input/output parameters (the style guide recommends mutable references in some cases).

Our continuous integration builds on GitHub Actions run the unit test suites on a variety of platforms and configuration, including using Address Sanitizer and Undefined Behavior Sanitizer to check for various patterns of misbehaviour such as memory leaks. In addition, the codebase is subjected to a number of code style and code cleanliness checks.

In order to have a passing CI build, your modified Git branch must pass the following checks:

  • C++ builds with the project's active version of clang without compiler warnings with -DBUILD_WARNING_LEVEL=CHECKIN. Note that there are classes of warnings (such as -Wdocumentation, see more on this below) that are not caught by gcc.
  • Passes various C++ (and others) style checks by running pre-commit run --show-diff-on-failure --color=always --all-files cpp.

On pull requests, the "Dev / Lint" pipeline will run these checks, and report what files/lines need to be fixed, if any.

Checking for ABI and API stability

To build ABI compliance reports, you need to install the two tools abi-dumper and abi-compliance-checker.

Build Arrow C++ in Debug mode, alternatively you could use -Og which also builds with the necessary symbols but includes a bit of code optimization. Once the build has finished, you can generate ABI reports using:

abi-dumper -lver 9 debug/libarrow.so -o ABI-9.dump

The above version number is freely selectable. As we want to compare versions, you should now git checkout the version you want to compare it to and re-run the above command using a different version number. Once both reports are generated, you can build a comparison report using

abi-compliance-checker -l libarrow -d1 ABI-PY-9.dump -d2 ABI-PY-10.dump

The report is then generated in compat_reports/libarrow as a HTML.

API Documentation

We use Doxygen style comments (///) in header files for comments that we wish to show up in API documentation for classes and functions.

When using clang and building with -DBUILD_WARNING_LEVEL=CHECKIN, the -Wdocumentation flag is used which checks for some common documentation inconsistencies, like documenting some, but not all function parameters with \param. See the LLVM documentation warnings section for more about this.

While we publish the API documentation as part of the main Sphinx-based documentation site, you can also build the C++ API documentation anytime using Doxygen. Run the following command from the cpp/apidoc directory:

doxygen Doxyfile

This requires Doxygen to be installed.

Apache Parquet Development

To build the C++ libraries for Apache Parquet, add the flag -DARROW_PARQUET=ON when invoking CMake. To build Apache Parquet with encryption support, add the flag -DPARQUET_REQUIRE_ENCRYPTION=ON when invoking CMake. The Parquet libraries and unit tests can be built with the parquet make target:

make parquet

On Linux and macOS if you do not have Apache Thrift installed on your system, or you are building with -DThrift_SOURCE=BUNDLED, you must install bison and flex packages. On Windows we handle these build dependencies automatically when building Thrift from source.

Running ctest -L unittest will run all built C++ unit tests, while ctest -L parquet will run only the Parquet unit tests. The unit tests depend on an environment variable PARQUET_TEST_DATA that depends on a git submodule to the repository https://github.com/apache/parquet-testing:

git submodule update --init
export PARQUET_TEST_DATA=$ARROW_ROOT/cpp/submodules/parquet-testing/data

Here $ARROW_ROOT is the absolute path to the Arrow codebase.

Arrow Flight RPC

In addition to the Arrow dependencies, Flight requires:

  • gRPC (>= 1.14, roughly)
  • Protobuf (>= 3.6, earlier versions may work)
  • c-ares (used by gRPC)

By default, Arrow will try to download and build these dependencies when building Flight.

The optional flight libraries and tests can be built by passing -DARROW_FLIGHT=ON.

cmake .. -DARROW_FLIGHT=ON -DARROW_BUILD_TESTS=ON
make

You can also use existing installations of the extra dependencies. When building, set the environment variables gRPC_ROOT and/or Protobuf_ROOT and/or c-ares_ROOT.

We are developing against recent versions of gRPC, and the versions. The libgrpc package available from https://conda-forge.org/ is one reliable way to obtain gRPC in a cross-platform way. You may try using system libraries for gRPC and Protobuf, but these are likely to be too old. On macOS, you can try Homebrew:

brew install grpc
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
development.rst
Latest commit
History
History
224 lines (162 loc) · 8.44 KB
development.rst
File metadata and controls
224 lines (162 loc) · 8.44 KB
Raw
Development Guidelines
This section provides information for developers who wish to contribute to the
C++ codebase.
Note
Since most of the project's developers work on Linux or macOS, not all
features or developer tools are uniformly supported on Windows. If you are
on Windows, have a look at :ref:`developers-cpp-windows`.
Compiler warning levels
The BUILD_WARNING_LEVEL CMake option switches between sets of predetermined
compiler warning levels that we use for code tidiness. For release builds, the
default warning level is PRODUCTION, while for debug builds the default is
CHECKIN.
When using CHECKIN for debug builds, -Werror is added when using gcc
and clang, causing build failures for any warning, and /WX is set with MSVC
having the same effect.
Running unit tests
The -DARROW_BUILD_TESTS=ON CMake option enables building of unit test
executables. You can then either run them individually, by launching the
desired executable, or run them all at once by launching the ctest
executable (which is part of the CMake suite).
A possible invocation is something like:
$ ctest -j16 --output-on-failure
where the -j16 option runs up to 16 tests in parallel, taking advantage
of multiple CPU cores and hardware threads.
Running benchmarks
The -DARROW_BUILD_BENCHMARKS=ON CMake option enables building of benchmark
executables. You can then run benchmarks individually by launching the
corresponding executable from the command line, e.g.:
$ ./build/release/arrow-builder-benchmark
Note
For meaningful benchmark numbers, it is very strongly recommended to build
in Release mode, so as to enable compiler optimizations.
Code Style, Linting, and CI
This project follows Google's C++ Style Guide with these exceptions:
  • We relax the line length restriction to 90 characters.
  • We use the NULLPTR macro in header files (instead of nullptr) defined
in src/arrow/util/macros.h to support building C++/CLI (ARROW-1134).
  • We relax the guide's rules regarding structs. For public headers we should
use struct only for objects that are principally simple data containers where
it is OK to expose all the internal members and any methods are primarily
conveniences. For private headers the rules are relaxed further and structs
can be used where convenient for types that do not need access control even
though they may not be simple data containers.
  • We prefer pointers for output and input/output parameters (the
style guide recommends mutable references in some cases).
Our continuous integration builds on GitHub Actions run the unit test
suites on a variety of platforms and configuration, including using
Address Sanitizer and Undefined Behavior Sanitizer to check for various
patterns of misbehaviour such as memory leaks. In addition, the
codebase is subjected to a number of code style and code cleanliness checks.
In order to have a passing CI build, your modified Git branch must pass the
following checks:
  • C++ builds with the project's active version of clang without
compiler warnings with -DBUILD_WARNING_LEVEL=CHECKIN. Note that
there are classes of warnings (such as -Wdocumentation, see more
on this below) that are not caught by gcc.
  • Passes various C++ (and others) style checks by running
pre-commit run --show-diff-on-failure --color=always --all-files
cpp.
On pull requests, the "Dev / Lint" pipeline will run these checks, and report
what files/lines need to be fixed, if any.
Checking for ABI and API stability
To build ABI compliance reports, you need to install the two tools
abi-dumper and abi-compliance-checker.
Build Arrow C++ in Debug mode, alternatively you could use -Og which also
builds with the necessary symbols but includes a bit of code optimization.
Once the build has finished, you can generate ABI reports using:
abi-dumper -lver 9 debug/libarrow.so -o ABI-9.dump
The above version number is freely selectable. As we want to compare versions,
you should now git checkout the version you want to compare it to and re-run
the above command using a different version number. Once both reports are
generated, you can build a comparison report using
abi-compliance-checker -l libarrow -d1 ABI-PY-9.dump -d2 ABI-PY-10.dump
The report is then generated in compat_reports/libarrow as a HTML.
API Documentation
We use Doxygen style comments (///) in header files for comments
that we wish to show up in API documentation for classes and
functions.
When using clang and building with
-DBUILD_WARNING_LEVEL=CHECKIN, the -Wdocumentation flag is
used which checks for some common documentation inconsistencies, like
documenting some, but not all function parameters with \param. See
the LLVM documentation warnings section
for more about this.
While we publish the API documentation as part of the main Sphinx-based
documentation site, you can also build the C++ API documentation anytime using
Doxygen. Run the following command from the cpp/apidoc directory:
doxygen Doxyfile
This requires Doxygen to be installed.
Apache Parquet Development
To build the C++ libraries for Apache Parquet, add the flag
-DARROW_PARQUET=ON when invoking CMake.
To build Apache Parquet with encryption support, add the flag
-DPARQUET_REQUIRE_ENCRYPTION=ON when invoking CMake. The Parquet libraries and unit tests
can be built with the parquet make target:
make parquet
On Linux and macOS if you do not have Apache Thrift installed on your system,
or you are building with -DThrift_SOURCE=BUNDLED, you must install
bison and flex packages. On Windows we handle these build dependencies
automatically when building Thrift from source.
Running ctest -L unittest will run all built C++ unit tests, while ctest -L
parquet will run only the Parquet unit tests. The unit tests depend on an
environment variable PARQUET_TEST_DATA that depends on a git submodule to the
repository https://github.com/apache/parquet-testing:
git submodule update --init
export PARQUET_TEST_DATA=$ARROW_ROOT/cpp/submodules/parquet-testing/data
Here $ARROW_ROOT is the absolute path to the Arrow codebase.
Arrow Flight RPC
In addition to the Arrow dependencies, Flight requires:
  • gRPC (>= 1.14, roughly)
  • Protobuf (>= 3.6, earlier versions may work)
  • c-ares (used by gRPC)
By default, Arrow will try to download and build these dependencies
when building Flight.
The optional flight libraries and tests can be built by passing
-DARROW_FLIGHT=ON.
cmake .. -DARROW_FLIGHT=ON -DARROW_BUILD_TESTS=ON
make
You can also use existing installations of the extra dependencies.
When building, set the environment variables gRPC_ROOT and/or
Protobuf_ROOT and/or c-ares_ROOT.
We are developing against recent versions of gRPC, and the versions. The
libgrpc package available from https://conda-forge.org/ is one reliable
way to obtain gRPC in a cross-platform way. You may try using system libraries
for gRPC and Protobuf, but these are likely to be too old. On macOS, you can
try Homebrew:
brew install grpc

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
development.rst
Latest commit
History
History
224 lines (162 loc) · 8.44 KB
development.rst
File metadata and controls
224 lines (162 loc) · 8.44 KB
Raw
Development Guidelines
This section provides information for developers who wish to contribute to the
C++ codebase.
Note
Since most of the project's developers work on Linux or macOS, not all
features or developer tools are uniformly supported on Windows. If you are
on Windows, have a look at :ref:`developers-cpp-windows`.
Compiler warning levels
The BUILD_WARNING_LEVEL CMake option switches between sets of predetermined
compiler warning levels that we use for code tidiness. For release builds, the
default warning level is PRODUCTION, while for debug builds the default is
CHECKIN.
When using CHECKIN for debug builds, -Werror is added when using gcc
and clang, causing build failures for any warning, and /WX is set with MSVC
having the same effect.
Running unit tests
The -DARROW_BUILD_TESTS=ON CMake option enables building of unit test
executables. You can then either run them individually, by launching the
desired executable, or run them all at once by launching the ctest
executable (which is part of the CMake suite).
A possible invocation is something like:
$ ctest -j16 --output-on-failure
where the -j16 option runs up to 16 tests in parallel, taking advantage
of multiple CPU cores and hardware threads.
Running benchmarks
The -DARROW_BUILD_BENCHMARKS=ON CMake option enables building of benchmark
executables. You can then run benchmarks individually by launching the
corresponding executable from the command line, e.g.:
$ ./build/release/arrow-builder-benchmark
Note
For meaningful benchmark numbers, it is very strongly recommended to build
in Release mode, so as to enable compiler optimizations.
Code Style, Linting, and CI
This project follows Google's C++ Style Guide with these exceptions:
  • We relax the line length restriction to 90 characters.
  • We use the NULLPTR macro in header files (instead of nullptr) defined
in src/arrow/util/macros.h to support building C++/CLI (ARROW-1134).
  • We relax the guide's rules regarding structs. For public headers we should
use struct only for objects that are principally simple data containers where
it is OK to expose all the internal members and any methods are primarily
conveniences. For private headers the rules are relaxed further and structs
can be used where convenient for types that do not need access control even
though they may not be simple data containers.
  • We prefer pointers for output and input/output parameters (the
style guide recommends mutable references in some cases).
Our continuous integration builds on GitHub Actions run the unit test
suites on a variety of platforms and configuration, including using
Address Sanitizer and Undefined Behavior Sanitizer to check for various
patterns of misbehaviour such as memory leaks. In addition, the
codebase is subjected to a number of code style and code cleanliness checks.
In order to have a passing CI build, your modified Git branch must pass the
following checks:
  • C++ builds with the project's active version of clang without
compiler warnings with -DBUILD_WARNING_LEVEL=CHECKIN. Note that
there are classes of warnings (such as -Wdocumentation, see more
on this below) that are not caught by gcc.
  • Passes various C++ (and others) style checks by running
pre-commit run --show-diff-on-failure --color=always --all-files
cpp.
On pull requests, the "Dev / Lint" pipeline will run these checks, and report
what files/lines need to be fixed, if any.
Checking for ABI and API stability
To build ABI compliance reports, you need to install the two tools
abi-dumper and abi-compliance-checker.
Build Arrow C++ in Debug mode, alternatively you could use -Og which also
builds with the necessary symbols but includes a bit of code optimization.
Once the build has finished, you can generate ABI reports using:
abi-dumper -lver 9 debug/libarrow.so -o ABI-9.dump
The above version number is freely selectable. As we want to compare versions,
you should now git checkout the version you want to compare it to and re-run
the above command using a different version number. Once both reports are
generated, you can build a comparison report using
abi-compliance-checker -l libarrow -d1 ABI-PY-9.dump -d2 ABI-PY-10.dump
The report is then generated in compat_reports/libarrow as a HTML.
API Documentation
We use Doxygen style comments (///) in header files for comments
that we wish to show up in API documentation for classes and
functions.
When using clang and building with
-DBUILD_WARNING_LEVEL=CHECKIN, the -Wdocumentation flag is
used which checks for some common documentation inconsistencies, like
documenting some, but not all function parameters with \param. See
the LLVM documentation warnings section
for more about this.
While we publish the API documentation as part of the main Sphinx-based
documentation site, you can also build the C++ API documentation anytime using
Doxygen. Run the following command from the cpp/apidoc directory:
doxygen Doxyfile
This requires Doxygen to be installed.
Apache Parquet Development
To build the C++ libraries for Apache Parquet, add the flag
-DARROW_PARQUET=ON when invoking CMake.
To build Apache Parquet with encryption support, add the flag
-DPARQUET_REQUIRE_ENCRYPTION=ON when invoking CMake. The Parquet libraries and unit tests
can be built with the parquet make target:
make parquet
On Linux and macOS if you do not have Apache Thrift installed on your system,
or you are building with -DThrift_SOURCE=BUNDLED, you must install
bison and flex packages. On Windows we handle these build dependencies
automatically when building Thrift from source.
Running ctest -L unittest will run all built C++ unit tests, while ctest -L
parquet will run only the Parquet unit tests. The unit tests depend on an
environment variable PARQUET_TEST_DATA that depends on a git submodule to the
repository https://github.com/apache/parquet-testing:
git submodule update --init
export PARQUET_TEST_DATA=$ARROW_ROOT/cpp/submodules/parquet-testing/data
Here $ARROW_ROOT is the absolute path to the Arrow codebase.
Arrow Flight RPC
In addition to the Arrow dependencies, Flight requires:
  • gRPC (>= 1.14, roughly)
  • Protobuf (>= 3.6, earlier versions may work)
  • c-ares (used by gRPC)
By default, Arrow will try to download and build these dependencies
when building Flight.
The optional flight libraries and tests can be built by passing
-DARROW_FLIGHT=ON.
cmake .. -DARROW_FLIGHT=ON -DARROW_BUILD_TESTS=ON
make
You can also use existing installations of the extra dependencies.
When building, set the environment variables gRPC_ROOT and/or
Protobuf_ROOT and/or c-ares_ROOT.
We are developing against recent versions of gRPC, and the versions. The
libgrpc package available from https://conda-forge.org/ is one reliable
way to obtain gRPC in a cross-platform way. You may try using system libraries
for gRPC and Protobuf, but these are likely to be too old. On macOS, you can
try Homebrew:
brew install grpc

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
development.rst
Latest commit
History
History
224 lines (162 loc) · 8.44 KB
development.rst
File metadata and controls
224 lines (162 loc) · 8.44 KB
Raw
Development Guidelines
This section provides information for developers who wish to contribute to the
C++ codebase.
Note
Since most of the project's developers work on Linux or macOS, not all
features or developer tools are uniformly supported on Windows. If you are
on Windows, have a look at :ref:`developers-cpp-windows`.
Compiler warning levels
The BUILD_WARNING_LEVEL CMake option switches between sets of predetermined
compiler warning levels that we use for code tidiness. For release builds, the
default warning level is PRODUCTION, while for debug builds the default is
CHECKIN.
When using CHECKIN for debug builds, -Werror is added when using gcc
and clang, causing build failures for any warning, and /WX is set with MSVC
having the same effect.
Running unit tests
The -DARROW_BUILD_TESTS=ON CMake option enables building of unit test
executables. You can then either run them individually, by launching the
desired executable, or run them all at once by launching the ctest
executable (which is part of the CMake suite).
A possible invocation is something like:
$ ctest -j16 --output-on-failure
where the -j16 option runs up to 16 tests in parallel, taking advantage
of multiple CPU cores and hardware threads.
Running benchmarks
The -DARROW_BUILD_BENCHMARKS=ON CMake option enables building of benchmark
executables. You can then run benchmarks individually by launching the
corresponding executable from the command line, e.g.:
$ ./build/release/arrow-builder-benchmark
Note
For meaningful benchmark numbers, it is very strongly recommended to build
in Release mode, so as to enable compiler optimizations.
Code Style, Linting, and CI
This project follows Google's C++ Style Guide with these exceptions:
  • We relax the line length restriction to 90 characters.
  • We use the NULLPTR macro in header files (instead of nullptr) defined
in src/arrow/util/macros.h to support building C++/CLI (ARROW-1134).
  • We relax the guide's rules regarding structs. For public headers we should
use struct only for objects that are principally simple data containers where
it is OK to expose all the internal members and any methods are primarily
conveniences. For private headers the rules are relaxed further and structs
can be used where convenient for types that do not need access control even
though they may not be simple data containers.
  • We prefer pointers for output and input/output parameters (the
style guide recommends mutable references in some cases).
Our continuous integration builds on GitHub Actions run the unit test
suites on a variety of platforms and configuration, including using
Address Sanitizer and Undefined Behavior Sanitizer to check for various
patterns of misbehaviour such as memory leaks. In addition, the
codebase is subjected to a number of code style and code cleanliness checks.
In order to have a passing CI build, your modified Git branch must pass the
following checks:
  • C++ builds with the project's active version of clang without
compiler warnings with -DBUILD_WARNING_LEVEL=CHECKIN. Note that
there are classes of warnings (such as -Wdocumentation, see more
on this below) that are not caught by gcc.
  • Passes various C++ (and others) style checks by running
pre-commit run --show-diff-on-failure --color=always --all-files
cpp.
On pull requests, the "Dev / Lint" pipeline will run these checks, and report
what files/lines need to be fixed, if any.
Checking for ABI and API stability
To build ABI compliance reports, you need to install the two tools
abi-dumper and abi-compliance-checker.
Build Arrow C++ in Debug mode, alternatively you could use -Og which also
builds with the necessary symbols but includes a bit of code optimization.
Once the build has finished, you can generate ABI reports using:
abi-dumper -lver 9 debug/libarrow.so -o ABI-9.dump
The above version number is freely selectable. As we want to compare versions,
you should now git checkout the version you want to compare it to and re-run
the above command using a different version number. Once both reports are
generated, you can build a comparison report using
abi-compliance-checker -l libarrow -d1 ABI-PY-9.dump -d2 ABI-PY-10.dump
The report is then generated in compat_reports/libarrow as a HTML.
API Documentation
We use Doxygen style comments (///) in header files for comments
that we wish to show up in API documentation for classes and
functions.
When using clang and building with
-DBUILD_WARNING_LEVEL=CHECKIN, the -Wdocumentation flag is
used which checks for some common documentation inconsistencies, like
documenting some, but not all function parameters with \param. See
the LLVM documentation warnings section
for more about this.
While we publish the API documentation as part of the main Sphinx-based
documentation site, you can also build the C++ API documentation anytime using
Doxygen. Run the following command from the cpp/apidoc directory:
doxygen Doxyfile
This requires Doxygen to be installed.
Apache Parquet Development
To build the C++ libraries for Apache Parquet, add the flag
-DARROW_PARQUET=ON when invoking CMake.
To build Apache Parquet with encryption support, add the flag
-DPARQUET_REQUIRE_ENCRYPTION=ON when invoking CMake. The Parquet libraries and unit tests
can be built with the parquet make target:
make parquet
On Linux and macOS if you do not have Apache Thrift installed on your system,
or you are building with -DThrift_SOURCE=BUNDLED, you must install
bison and flex packages. On Windows we handle these build dependencies
automatically when building Thrift from source.
Running ctest -L unittest will run all built C++ unit tests, while ctest -L
parquet will run only the Parquet unit tests. The unit tests depend on an
environment variable PARQUET_TEST_DATA that depends on a git submodule to the
repository https://github.com/apache/parquet-testing:
git submodule update --init
export PARQUET_TEST_DATA=$ARROW_ROOT/cpp/submodules/parquet-testing/data
Here $ARROW_ROOT is the absolute path to the Arrow codebase.
Arrow Flight RPC
In addition to the Arrow dependencies, Flight requires:
  • gRPC (>= 1.14, roughly)
  • Protobuf (>= 3.6, earlier versions may work)
  • c-ares (used by gRPC)
By default, Arrow will try to download and build these dependencies
when building Flight.
The optional flight libraries and tests can be built by passing
-DARROW_FLIGHT=ON.
cmake .. -DARROW_FLIGHT=ON -DARROW_BUILD_TESTS=ON
make
You can also use existing installations of the extra dependencies.
When building, set the environment variables gRPC_ROOT and/or
Protobuf_ROOT and/or c-ares_ROOT.
We are developing against recent versions of gRPC, and the versions. The
libgrpc package available from https://conda-forge.org/ is one reliable
way to obtain gRPC in a cross-platform way. You may try using system libraries
for gRPC and Protobuf, but these are likely to be too old. On macOS, you can
try Homebrew:
brew install grpc

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
development.rst
Latest commit
History
History
224 lines (162 loc) · 8.44 KB
development.rst
File metadata and controls
224 lines (162 loc) · 8.44 KB
Raw
Development Guidelines
This section provides information for developers who wish to contribute to the
C++ codebase.
Note
Since most of the project's developers work on Linux or macOS, not all
features or developer tools are uniformly supported on Windows. If you are
on Windows, have a look at :ref:`developers-cpp-windows`.
Compiler warning levels
The BUILD_WARNING_LEVEL CMake option switches between sets of predetermined
compiler warning levels that we use for code tidiness. For release builds, the
default warning level is PRODUCTION, while for debug builds the default is
CHECKIN.
When using CHECKIN for debug builds, -Werror is added when using gcc
and clang, causing build failures for any warning, and /WX is set with MSVC
having the same effect.
Running unit tests
The -DARROW_BUILD_TESTS=ON CMake option enables building of unit test
executables. You can then either run them individually, by launching the
desired executable, or run them all at once by launching the ctest
executable (which is part of the CMake suite).
A possible invocation is something like:
$ ctest -j16 --output-on-failure
where the -j16 option runs up to 16 tests in parallel, taking advantage
of multiple CPU cores and hardware threads.
Running benchmarks
The -DARROW_BUILD_BENCHMARKS=ON CMake option enables building of benchmark
executables. You can then run benchmarks individually by launching the
corresponding executable from the command line, e.g.:
$ ./build/release/arrow-builder-benchmark
Note
For meaningful benchmark numbers, it is very strongly recommended to build
in Release mode, so as to enable compiler optimizations.
Code Style, Linting, and CI
This project follows Google's C++ Style Guide with these exceptions:
  • We relax the line length restriction to 90 characters.
  • We use the NULLPTR macro in header files (instead of nullptr) defined
in src/arrow/util/macros.h to support building C++/CLI (ARROW-1134).
  • We relax the guide's rules regarding structs. For public headers we should
use struct only for objects that are principally simple data containers where
it is OK to expose all the internal members and any methods are primarily
conveniences. For private headers the rules are relaxed further and structs
can be used where convenient for types that do not need access control even
though they may not be simple data containers.
  • We prefer pointers for output and input/output parameters (the
style guide recommends mutable references in some cases).
Our continuous integration builds on GitHub Actions run the unit test
suites on a variety of platforms and configuration, including using
Address Sanitizer and Undefined Behavior Sanitizer to check for various
patterns of misbehaviour such as memory leaks. In addition, the
codebase is subjected to a number of code style and code cleanliness checks.
In order to have a passing CI build, your modified Git branch must pass the
following checks:
  • C++ builds with the project's active version of clang without
compiler warnings with -DBUILD_WARNING_LEVEL=CHECKIN. Note that
there are classes of warnings (such as -Wdocumentation, see more
on this below) that are not caught by gcc.
  • Passes various C++ (and others) style checks by running
pre-commit run --show-diff-on-failure --color=always --all-files
cpp.
On pull requests, the "Dev / Lint" pipeline will run these checks, and report
what files/lines need to be fixed, if any.
Checking for ABI and API stability
To build ABI compliance reports, you need to install the two tools
abi-dumper and abi-compliance-checker.
Build Arrow C++ in Debug mode, alternatively you could use -Og which also
builds with the necessary symbols but includes a bit of code optimization.
Once the build has finished, you can generate ABI reports using:
abi-dumper -lver 9 debug/libarrow.so -o ABI-9.dump
The above version number is freely selectable. As we want to compare versions,
you should now git checkout the version you want to compare it to and re-run
the above command using a different version number. Once both reports are
generated, you can build a comparison report using
abi-compliance-checker -l libarrow -d1 ABI-PY-9.dump -d2 ABI-PY-10.dump
The report is then generated in compat_reports/libarrow as a HTML.
API Documentation
We use Doxygen style comments (///) in header files for comments
that we wish to show up in API documentation for classes and
functions.
When using clang and building with
-DBUILD_WARNING_LEVEL=CHECKIN, the -Wdocumentation flag is
used which checks for some common documentation inconsistencies, like
documenting some, but not all function parameters with \param. See
the LLVM documentation warnings section
for more about this.
While we publish the API documentation as part of the main Sphinx-based
documentation site, you can also build the C++ API documentation anytime using
Doxygen. Run the following command from the cpp/apidoc directory:
doxygen Doxyfile
This requires Doxygen to be installed.
Apache Parquet Development
To build the C++ libraries for Apache Parquet, add the flag
-DARROW_PARQUET=ON when invoking CMake.
To build Apache Parquet with encryption support, add the flag
-DPARQUET_REQUIRE_ENCRYPTION=ON when invoking CMake. The Parquet libraries and unit tests
can be built with the parquet make target:
make parquet
On Linux and macOS if you do not have Apache Thrift installed on your system,
or you are building with -DThrift_SOURCE=BUNDLED, you must install
bison and flex packages. On Windows we handle these build dependencies
automatically when building Thrift from source.
Running ctest -L unittest will run all built C++ unit tests, while ctest -L
parquet will run only the Parquet unit tests. The unit tests depend on an
environment variable PARQUET_TEST_DATA that depends on a git submodule to the
repository https://github.com/apache/parquet-testing:
git submodule update --init
export PARQUET_TEST_DATA=$ARROW_ROOT/cpp/submodules/parquet-testing/data
Here $ARROW_ROOT is the absolute path to the Arrow codebase.
Arrow Flight RPC
In addition to the Arrow dependencies, Flight requires:
  • gRPC (>= 1.14, roughly)
  • Protobuf (>= 3.6, earlier versions may work)
  • c-ares (used by gRPC)
By default, Arrow will try to download and build these dependencies
when building Flight.
The optional flight libraries and tests can be built by passing
-DARROW_FLIGHT=ON.
cmake .. -DARROW_FLIGHT=ON -DARROW_BUILD_TESTS=ON
make
You can also use existing installations of the extra dependencies.
When building, set the environment variables gRPC_ROOT and/or
Protobuf_ROOT and/or c-ares_ROOT.
We are developing against recent versions of gRPC, and the versions. The
libgrpc package available from https://conda-forge.org/ is one reliable
way to obtain gRPC in a cross-platform way. You may try using system libraries
for gRPC and Protobuf, but these are likely to be too old. On macOS, you can
try Homebrew:
brew install grpc

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
development.rst
Latest commit
History
History
224 lines (162 loc) · 8.44 KB
development.rst
File metadata and controls
224 lines (162 loc) · 8.44 KB
Raw
Development Guidelines
This section provides information for developers who wish to contribute to the
C++ codebase.
Note
Since most of the project's developers work on Linux or macOS, not all
features or developer tools are uniformly supported on Windows. If you are
on Windows, have a look at :ref:`developers-cpp-windows`.
Compiler warning levels
The BUILD_WARNING_LEVEL CMake option switches between sets of predetermined
compiler warning levels that we use for code tidiness. For release builds, the
default warning level is PRODUCTION, while for debug builds the default is
CHECKIN.
When using CHECKIN for debug builds, -Werror is added when using gcc
and clang, causing build failures for any warning, and /WX is set with MSVC
having the same effect.
Running unit tests
The -DARROW_BUILD_TESTS=ON CMake option enables building of unit test
executables. You can then either run them individually, by launching the
desired executable, or run them all at once by launching the ctest
executable (which is part of the CMake suite).
A possible invocation is something like:
$ ctest -j16 --output-on-failure
where the -j16 option runs up to 16 tests in parallel, taking advantage
of multiple CPU cores and hardware threads.
Running benchmarks
The -DARROW_BUILD_BENCHMARKS=ON CMake option enables building of benchmark
executables. You can then run benchmarks individually by launching the
corresponding executable from the command line, e.g.:
$ ./build/release/arrow-builder-benchmark
Note
For meaningful benchmark numbers, it is very strongly recommended to build
in Release mode, so as to enable compiler optimizations.
Code Style, Linting, and CI
This project follows Google's C++ Style Guide with these exceptions:
  • We relax the line length restriction to 90 characters.
  • We use the NULLPTR macro in header files (instead of nullptr) defined
in src/arrow/util/macros.h to support building C++/CLI (ARROW-1134).
  • We relax the guide's rules regarding structs. For public headers we should
use struct only for objects that are principally simple data containers where
it is OK to expose all the internal members and any methods are primarily
conveniences. For private headers the rules are relaxed further and structs
can be used where convenient for types that do not need access control even
though they may not be simple data containers.
  • We prefer pointers for output and input/output parameters (the
style guide recommends mutable references in some cases).
Our continuous integration builds on GitHub Actions run the unit test
suites on a variety of platforms and configuration, including using
Address Sanitizer and Undefined Behavior Sanitizer to check for various
patterns of misbehaviour such as memory leaks. In addition, the
codebase is subjected to a number of code style and code cleanliness checks.
In order to have a passing CI build, your modified Git branch must pass the
following checks:
  • C++ builds with the project's active version of clang without
compiler warnings with -DBUILD_WARNING_LEVEL=CHECKIN. Note that
there are classes of warnings (such as -Wdocumentation, see more
on this below) that are not caught by gcc.
  • Passes various C++ (and others) style checks by running
pre-commit run --show-diff-on-failure --color=always --all-files
cpp.
On pull requests, the "Dev / Lint" pipeline will run these checks, and report
what files/lines need to be fixed, if any.
Checking for ABI and API stability
To build ABI compliance reports, you need to install the two tools
abi-dumper and abi-compliance-checker.
Build Arrow C++ in Debug mode, alternatively you could use -Og which also
builds with the necessary symbols but includes a bit of code optimization.
Once the build has finished, you can generate ABI reports using:
abi-dumper -lver 9 debug/libarrow.so -o ABI-9.dump
The above version number is freely selectable. As we want to compare versions,
you should now git checkout the version you want to compare it to and re-run
the above command using a different version number. Once both reports are
generated, you can build a comparison report using
abi-compliance-checker -l libarrow -d1 ABI-PY-9.dump -d2 ABI-PY-10.dump
The report is then generated in compat_reports/libarrow as a HTML.
API Documentation
We use Doxygen style comments (///) in header files for comments
that we wish to show up in API documentation for classes and
functions.
When using clang and building with
-DBUILD_WARNING_LEVEL=CHECKIN, the -Wdocumentation flag is
used which checks for some common documentation inconsistencies, like
documenting some, but not all function parameters with \param. See
the LLVM documentation warnings section
for more about this.
While we publish the API documentation as part of the main Sphinx-based
documentation site, you can also build the C++ API documentation anytime using
Doxygen. Run the following command from the cpp/apidoc directory:
doxygen Doxyfile
This requires Doxygen to be installed.
Apache Parquet Development
To build the C++ libraries for Apache Parquet, add the flag
-DARROW_PARQUET=ON when invoking CMake.
To build Apache Parquet with encryption support, add the flag
-DPARQUET_REQUIRE_ENCRYPTION=ON when invoking CMake. The Parquet libraries and unit tests
can be built with the parquet make target:
make parquet
On Linux and macOS if you do not have Apache Thrift installed on your system,
or you are building with -DThrift_SOURCE=BUNDLED, you must install
bison and flex packages. On Windows we handle these build dependencies
automatically when building Thrift from source.
Running ctest -L unittest will run all built C++ unit tests, while ctest -L
parquet will run only the Parquet unit tests. The unit tests depend on an
environment variable PARQUET_TEST_DATA that depends on a git submodule to the
repository https://github.com/apache/parquet-testing:
git submodule update --init
export PARQUET_TEST_DATA=$ARROW_ROOT/cpp/submodules/parquet-testing/data
Here $ARROW_ROOT is the absolute path to the Arrow codebase.
Arrow Flight RPC
In addition to the Arrow dependencies, Flight requires:
  • gRPC (>= 1.14, roughly)
  • Protobuf (>= 3.6, earlier versions may work)
  • c-ares (used by gRPC)
By default, Arrow will try to download and build these dependencies
when building Flight.
The optional flight libraries and tests can be built by passing
-DARROW_FLIGHT=ON.
cmake .. -DARROW_FLIGHT=ON -DARROW_BUILD_TESTS=ON
make
You can also use existing installations of the extra dependencies.
When building, set the environment variables gRPC_ROOT and/or
Protobuf_ROOT and/or c-ares_ROOT.
We are developing against recent versions of gRPC, and the versions. The
libgrpc package available from https://conda-forge.org/ is one reliable
way to obtain gRPC in a cross-platform way. You may try using system libraries
for gRPC and Protobuf, but these are likely to be too old. On macOS, you can
try Homebrew:
brew install grpc

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
development.rst
Latest commit
History
History
224 lines (162 loc) · 8.44 KB
development.rst
File metadata and controls
224 lines (162 loc) · 8.44 KB
Raw
Development Guidelines
This section provides information for developers who wish to contribute to the
C++ codebase.
Note
Since most of the project's developers work on Linux or macOS, not all
features or developer tools are uniformly supported on Windows. If you are
on Windows, have a look at :ref:`developers-cpp-windows`.
Compiler warning levels
The BUILD_WARNING_LEVEL CMake option switches between sets of predetermined
compiler warning levels that we use for code tidiness. For release builds, the
default warning level is PRODUCTION, while for debug builds the default is
CHECKIN.
When using CHECKIN for debug builds, -Werror is added when using gcc
and clang, causing build failures for any warning, and /WX is set with MSVC
having the same effect.
Running unit tests
The -DARROW_BUILD_TESTS=ON CMake option enables building of unit test
executables. You can then either run them individually, by launching the
desired executable, or run them all at once by launching the ctest
executable (which is part of the CMake suite).
A possible invocation is something like:
$ ctest -j16 --output-on-failure
where the -j16 option runs up to 16 tests in parallel, taking advantage
of multiple CPU cores and hardware threads.
Running benchmarks
The -DARROW_BUILD_BENCHMARKS=ON CMake option enables building of benchmark
executables. You can then run benchmarks individually by launching the
corresponding executable from the command line, e.g.:
$ ./build/release/arrow-builder-benchmark
Note
For meaningful benchmark numbers, it is very strongly recommended to build
in Release mode, so as to enable compiler optimizations.
Code Style, Linting, and CI
This project follows Google's C++ Style Guide with these exceptions:
  • We relax the line length restriction to 90 characters.
  • We use the NULLPTR macro in header files (instead of nullptr) defined
in src/arrow/util/macros.h to support building C++/CLI (ARROW-1134).
  • We relax the guide's rules regarding structs. For public headers we should
use struct only for objects that are principally simple data containers where
it is OK to expose all the internal members and any methods are primarily
conveniences. For private headers the rules are relaxed further and structs
can be used where convenient for types that do not need access control even
though they may not be simple data containers.
  • We prefer pointers for output and input/output parameters (the
style guide recommends mutable references in some cases).
Our continuous integration builds on GitHub Actions run the unit test
suites on a variety of platforms and configuration, including using
Address Sanitizer and Undefined Behavior Sanitizer to check for various
patterns of misbehaviour such as memory leaks. In addition, the
codebase is subjected to a number of code style and code cleanliness checks.
In order to have a passing CI build, your modified Git branch must pass the
following checks:
  • C++ builds with the project's active version of clang without
compiler warnings with -DBUILD_WARNING_LEVEL=CHECKIN. Note that
there are classes of warnings (such as -Wdocumentation, see more
on this below) that are not caught by gcc.
  • Passes various C++ (and others) style checks by running
pre-commit run --show-diff-on-failure --color=always --all-files
cpp.
On pull requests, the "Dev / Lint" pipeline will run these checks, and report
what files/lines need to be fixed, if any.
Checking for ABI and API stability
To build ABI compliance reports, you need to install the two tools
abi-dumper and abi-compliance-checker.
Build Arrow C++ in Debug mode, alternatively you could use -Og which also
builds with the necessary symbols but includes a bit of code optimization.
Once the build has finished, you can generate ABI reports using:
abi-dumper -lver 9 debug/libarrow.so -o ABI-9.dump
The above version number is freely selectable. As we want to compare versions,
you should now git checkout the version you want to compare it to and re-run
the above command using a different version number. Once both reports are
generated, you can build a comparison report using
abi-compliance-checker -l libarrow -d1 ABI-PY-9.dump -d2 ABI-PY-10.dump
The report is then generated in compat_reports/libarrow as a HTML.
API Documentation
We use Doxygen style comments (///) in header files for comments
that we wish to show up in API documentation for classes and
functions.
When using clang and building with
-DBUILD_WARNING_LEVEL=CHECKIN, the -Wdocumentation flag is
used which checks for some common documentation inconsistencies, like
documenting some, but not all function parameters with \param. See
the LLVM documentation warnings section
for more about this.
While we publish the API documentation as part of the main Sphinx-based
documentation site, you can also build the C++ API documentation anytime using
Doxygen. Run the following command from the cpp/apidoc directory:
doxygen Doxyfile
This requires Doxygen to be installed.
Apache Parquet Development
To build the C++ libraries for Apache Parquet, add the flag
-DARROW_PARQUET=ON when invoking CMake.
To build Apache Parquet with encryption support, add the flag
-DPARQUET_REQUIRE_ENCRYPTION=ON when invoking CMake. The Parquet libraries and unit tests
can be built with the parquet make target:
make parquet
On Linux and macOS if you do not have Apache Thrift installed on your system,
or you are building with -DThrift_SOURCE=BUNDLED, you must install
bison and flex packages. On Windows we handle these build dependencies
automatically when building Thrift from source.
Running ctest -L unittest will run all built C++ unit tests, while ctest -L
parquet will run only the Parquet unit tests. The unit tests depend on an
environment variable PARQUET_TEST_DATA that depends on a git submodule to the
repository https://github.com/apache/parquet-testing:
git submodule update --init
export PARQUET_TEST_DATA=$ARROW_ROOT/cpp/submodules/parquet-testing/data
Here $ARROW_ROOT is the absolute path to the Arrow codebase.
Arrow Flight RPC
In addition to the Arrow dependencies, Flight requires:
  • gRPC (>= 1.14, roughly)
  • Protobuf (>= 3.6, earlier versions may work)
  • c-ares (used by gRPC)
By default, Arrow will try to download and build these dependencies
when building Flight.
The optional flight libraries and tests can be built by passing
-DARROW_FLIGHT=ON.
cmake .. -DARROW_FLIGHT=ON -DARROW_BUILD_TESTS=ON
make
You can also use existing installations of the extra dependencies.
When building, set the environment variables gRPC_ROOT and/or
Protobuf_ROOT and/or c-ares_ROOT.
We are developing against recent versions of gRPC, and the versions. The
libgrpc package available from https://conda-forge.org/ is one reliable
way to obtain gRPC in a cross-platform way. You may try using system libraries
for gRPC and Protobuf, but these are likely to be too old. On macOS, you can
try Homebrew:
brew install grpc

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
development.rst
Latest commit
History
History
224 lines (162 loc) · 8.44 KB
development.rst
File metadata and controls
224 lines (162 loc) · 8.44 KB
Raw
Development Guidelines
This section provides information for developers who wish to contribute to the
C++ codebase.
Note
Since most of the project's developers work on Linux or macOS, not all
features or developer tools are uniformly supported on Windows. If you are
on Windows, have a look at :ref:`developers-cpp-windows`.
Compiler warning levels
The BUILD_WARNING_LEVEL CMake option switches between sets of predetermined
compiler warning levels that we use for code tidiness. For release builds, the
default warning level is PRODUCTION, while for debug builds the default is
CHECKIN.
When using CHECKIN for debug builds, -Werror is added when using gcc
and clang, causing build failures for any warning, and /WX is set with MSVC
having the same effect.
Running unit tests
The -DARROW_BUILD_TESTS=ON CMake option enables building of unit test
executables. You can then either run them individually, by launching the
desired executable, or run them all at once by launching the ctest
executable (which is part of the CMake suite).
A possible invocation is something like:
$ ctest -j16 --output-on-failure
where the -j16 option runs up to 16 tests in parallel, taking advantage
of multiple CPU cores and hardware threads.
Running benchmarks
The -DARROW_BUILD_BENCHMARKS=ON CMake option enables building of benchmark
executables. You can then run benchmarks individually by launching the
corresponding executable from the command line, e.g.:
$ ./build/release/arrow-builder-benchmark
Note
For meaningful benchmark numbers, it is very strongly recommended to build
in Release mode, so as to enable compiler optimizations.
Code Style, Linting, and CI
This project follows Google's C++ Style Guide with these exceptions:
  • We relax the line length restriction to 90 characters.
  • We use the NULLPTR macro in header files (instead of nullptr) defined
in src/arrow/util/macros.h to support building C++/CLI (ARROW-1134).
  • We relax the guide's rules regarding structs. For public headers we should
use struct only for objects that are principally simple data containers where
it is OK to expose all the internal members and any methods are primarily
conveniences. For private headers the rules are relaxed further and structs
can be used where convenient for types that do not need access control even
though they may not be simple data containers.
  • We prefer pointers for output and input/output parameters (the
style guide recommends mutable references in some cases).
Our continuous integration builds on GitHub Actions run the unit test
suites on a variety of platforms and configuration, including using
Address Sanitizer and Undefined Behavior Sanitizer to check for various
patterns of misbehaviour such as memory leaks. In addition, the
codebase is subjected to a number of code style and code cleanliness checks.
In order to have a passing CI build, your modified Git branch must pass the
following checks:
  • C++ builds with the project's active version of clang without
compiler warnings with -DBUILD_WARNING_LEVEL=CHECKIN. Note that
there are classes of warnings (such as -Wdocumentation, see more
on this below) that are not caught by gcc.
  • Passes various C++ (and others) style checks by running
pre-commit run --show-diff-on-failure --color=always --all-files
cpp.
On pull requests, the "Dev / Lint" pipeline will run these checks, and report
what files/lines need to be fixed, if any.
Checking for ABI and API stability
To build ABI compliance reports, you need to install the two tools
abi-dumper and abi-compliance-checker.
Build Arrow C++ in Debug mode, alternatively you could use -Og which also
builds with the necessary symbols but includes a bit of code optimization.
Once the build has finished, you can generate ABI reports using:
abi-dumper -lver 9 debug/libarrow.so -o ABI-9.dump
The above version number is freely selectable. As we want to compare versions,
you should now git checkout the version you want to compare it to and re-run
the above command using a different version number. Once both reports are
generated, you can build a comparison report using
abi-compliance-checker -l libarrow -d1 ABI-PY-9.dump -d2 ABI-PY-10.dump
The report is then generated in compat_reports/libarrow as a HTML.
API Documentation
We use Doxygen style comments (///) in header files for comments
that we wish to show up in API documentation for classes and
functions.
When using clang and building with
-DBUILD_WARNING_LEVEL=CHECKIN, the -Wdocumentation flag is
used which checks for some common documentation inconsistencies, like
documenting some, but not all function parameters with \param. See
the LLVM documentation warnings section
for more about this.
While we publish the API documentation as part of the main Sphinx-based
documentation site, you can also build the C++ API documentation anytime using
Doxygen. Run the following command from the cpp/apidoc directory:
doxygen Doxyfile
This requires Doxygen to be installed.
Apache Parquet Development
To build the C++ libraries for Apache Parquet, add the flag
-DARROW_PARQUET=ON when invoking CMake.
To build Apache Parquet with encryption support, add the flag
-DPARQUET_REQUIRE_ENCRYPTION=ON when invoking CMake. The Parquet libraries and unit tests
can be built with the parquet make target:
make parquet
On Linux and macOS if you do not have Apache Thrift installed on your system,
or you are building with -DThrift_SOURCE=BUNDLED, you must install
bison and flex packages. On Windows we handle these build dependencies
automatically when building Thrift from source.
Running ctest -L unittest will run all built C++ unit tests, while ctest -L
parquet will run only the Parquet unit tests. The unit tests depend on an
environment variable PARQUET_TEST_DATA that depends on a git submodule to the
repository https://github.com/apache/parquet-testing:
git submodule update --init
export PARQUET_TEST_DATA=$ARROW_ROOT/cpp/submodules/parquet-testing/data
Here $ARROW_ROOT is the absolute path to the Arrow codebase.
Arrow Flight RPC
In addition to the Arrow dependencies, Flight requires:
  • gRPC (>= 1.14, roughly)
  • Protobuf (>= 3.6, earlier versions may work)
  • c-ares (used by gRPC)
By default, Arrow will try to download and build these dependencies
when building Flight.
The optional flight libraries and tests can be built by passing
-DARROW_FLIGHT=ON.
cmake .. -DARROW_FLIGHT=ON -DARROW_BUILD_TESTS=ON
make
You can also use existing installations of the extra dependencies.
When building, set the environment variables gRPC_ROOT and/or
Protobuf_ROOT and/or c-ares_ROOT.
We are developing against recent versions of gRPC, and the versions. The
libgrpc package available from https://conda-forge.org/ is one reliable
way to obtain gRPC in a cross-platform way. You may try using system libraries
for gRPC and Protobuf, but these are likely to be too old. On macOS, you can
try Homebrew:
brew install grpc