GH-37621: [Packaging][Conda] Sync conda recipes with feedstocks - #37624

Merged
jorisvandenbossche merged 9 commits into
apache:mainfrom
h-vetinari:conda_sync
Sep 28, 2023
Merged

GH-37621: [Packaging][Conda] Sync conda recipes with feedstocks#37624
jorisvandenbossche merged 9 commits into
apache:mainfrom
h-vetinari:conda_sync

Conversation

@h-vetinari

@h-vetinarih-vetinari commented Sep 8, 2023•
edited
Loading

Copy link
Copy Markdown
Contributor

Syncing after the release of 13.0.0 + a couple of migrations (state as of conda-forge/arrow-cpp-feedstock#1168 & conda-forge/r-arrow-feedstock#68)

Relevant updates:

Also some further hardening of the activation scripts & clean-ups for dependencies & test skips.

@github-actions

github-actionsBot commented Sep 8, 2023

Copy link
Copy Markdown

⚠️ GitHub issue #37621has been automatically assigned in GitHub to PR creator.

@github-actionsgithub-actionsBot added the awaiting review Awaiting review label Sep 8, 2023
@h-vetinari

h-vetinari commented Sep 8, 2023

Copy link
Copy Markdown
ContributorAuthor

@github-actions crossbow submit -g conda

@github-actions

github-actionsBot commented Sep 8, 2023

Copy link
Copy Markdown

Revision: c04c157

Submitted crossbow builds: ursacomputing/crossbow @ actions-630421cd79

TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cpu-r43Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cpu-r43Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-arm64-cpu-r43Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r42Azure
conda-osx-x64-cpu-r43Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure

@github-actionsgithub-actionsBot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Sep 8, 2023
@h-vetinari

h-vetinari commented Sep 8, 2023

Copy link
Copy Markdown
ContributorAuthor

Something is not working out with the CUDA cross-compilation here:

[82/139] Linking CXX shared module release/_cuda.cpython-310-powerpc64le-linux-gnu.so
FAILED: release/_cuda.cpython-310-powerpc64le-linux-gnu.so : && $BUILD_PREFIX/bin/powerpc64le-conda-linux-gnu-c++ -fPIC -Wno-noexcept-type -Wall -fno-semantic-interposition -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -mcpu=power8 -mtune=power8 -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -isystem $PREFIX/include -fdebug-prefix-map=/build/apache-arrow_1694161537567/work=/usr/local/src/conda/pyarrow-14.0.0.dev124 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -isystem /usr/local/cuda/targets/ppc64le-linux/include -fdiagnostics-color=always -fno-omit-frame-pointer -Wno-unused-variable -Wno-maybe-uninitialized -O3 -DNDEBUG -O2 -ftree-vectorize -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--allow-shlib-undefined -Wl,-rpath,$PREFIX/lib -Wl,-rpath-link,$PREFIX/lib -L$PREFIX/lib -shared -o release/_cuda.cpython-310-powerpc64le-linux-gnu.so CMakeFiles/_cuda.dir/_cuda.cpp.o -Wl,-rpath,"\$ORIGIN" release/libarrow_python.so $PREFIX/lib/libarrow_cuda.so.1400.0.0 $PREFIX/lib/libarrow_dataset.so.1400.0.0 $PREFIX/lib/libarrow_acero.so.1400.0.0 $PREFIX/lib/libparquet.so.1400.0.0 $PREFIX/lib/libarrow.so.1400.0.0 /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so && :
$BUILD_PREFIX/powerpc64le-conda-linux-gnu/bin/ld: /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status

It does work in conda-forge, and CUDA on aarch/ppc is a bit more of a niche setup, so I guess we could also drop it here. For completeness, I looked at the diff for the build scripts, but I couldn't determine something that would touch upon where the CUDA_HOME would point -- the error is pretty clearly that we're not pointing to the ppc version of libcuda.so:

$BUILD_PREFIX/powerpc64le-conda-linux-gnu/bin/ld: /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so: error adding symbols: file in wrong format

@h-vetinari

h-vetinari commented Sep 8, 2023

Copy link
Copy Markdown
ContributorAuthor

Is it intended that gandiva now has a run-time dependence on libllvm?

 import pyarrow.gandiva
ImportError: libLLVM-15.so: cannot open shared object file: No such file or directory

Adding the respective host-dependence (+ respective run-export) is not hard, I'm just double-checking that this is intentional.

CC @pitrou@kou@raulcd@jorisvandenbossche

PS. This is a rare case where windows looks better than unix, only one test failure:

=========================== short test summary info ===========================
FAILED tests/parquet/test_basic.py::test_fastparquet_cross_compatibility - As...
= 1 failed, 7154 passed, 355 skipped, 12 deselected, 22 xfailed, 2 xpassed, 8 warnings in 302.94s (0:05:02) =

@kou

kou commented Sep 8, 2023

Copy link
Copy Markdown
Member

Yes. It's caused by #37412.
We use libLLVM.so by default on non-Windows. If you don't like it you can add -DARROW_LLVM_USE_SHARED=OFF to use static LLVM libraries.
(I think that libLLVM.so is better to reduce duplication.)

@h-vetinarih-vetinari mentioned this pull request Sep 8, 2023
Closed
@h-vetinari

h-vetinari commented Sep 12, 2023

Copy link
Copy Markdown
ContributorAuthor

It does work in conda-forge, and CUDA on aarch/ppc is a bit more of a niche setup, so I guess we could also drop it here.

I started debugging this PR on conda-forge infrastructure in conda-forge/arrow-cpp-feedstock#1170, and it turns out that the problem hits us there as well. I'm not sure what changed since 13.0, but it seems something is now overriding (or not respecting) our CUDA_HOME, which is necessary to make cross-compilation work with CUDA...

@kou

kou commented Sep 13, 2023

Copy link
Copy Markdown
Member

Could you try -DCUDAToolkit_ROOT=${CUDA_HOME} instead of -DCUDA_TOOLKIT_ROOT_DIR=${CUDA_HOME}?

CUDA_TOOLKIT_ROOT_DIR is for old FindCUDA: https://cmake.org/cmake/help/latest/module/FindCUDA.html
We use FindCUDAToolkit for new CMake: https://cmake.org/cmake/help/latest/module/FindCUDAToolkit.html

@h-vetinari

h-vetinari commented Sep 13, 2023

Copy link
Copy Markdown
ContributorAuthor

Thanks for the quick response, that sounds like a very promising candidate! Rebased & retriggered conda-forge/arrow-cpp-feedstock#1170. Will sync back to this PR if passing.

@h-vetinarih-vetinari mentioned this pull request Sep 13, 2023
Closed
@github-actionsgithub-actionsBot added Component: Python awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Sep 17, 2023
h-vetinari
h-vetinari commented Sep 17, 2023
View reviewed changes
Comment threadpython/pyarrow/tests/test_fs.py Outdated
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
Comment on lines +389 to +390
# currently broken
{% set tests_to_skip = tests_to_skip + " or test_fastparquet_cross_compatibility" %}

@h-vetinarih-vetinariSep 17, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't raised an issue for this yet, but this is consistently failing. Not sure what's the difference between our CI and the one here.

@jorisvandenbosschejorisvandenbosscheSep 21, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know a CI build log that shows the error for this? (the last ones will have this already skipped)
(to have an idea what is going on here)

@h-vetinarih-vetinariSep 21, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

_____________________ test_fastparquet_cross_compatibility _____________________
tempdir = PosixPath('/tmp/pytest-of-conda/pytest-0/test_fastparquet_cross_compati0')
@pytest.mark.pandas
@pytest.mark.fastparquet
@pytest.mark.filterwarnings("ignore:RangeIndex:FutureWarning")
@pytest.mark.filterwarnings("ignore:tostring:DeprecationWarning:fastparquet")
def test_fastparquet_cross_compatibility(tempdir):
fp = pytest.importorskip('fastparquet')
df = pd.DataFrame(
{
"a": list("abc"),
"b": list(range(1, 4)),
"c": np.arange(4.0, 7.0, dtype="float64"),
"d": [True, False, True],
"e": pd.date_range("20130101", periods=3),
"f": pd.Categorical(["a", "b", "a"]),
# fastparquet writes list as BYTE_ARRAY JSON, so no roundtrip
# "g": [[1, 2], None, [1, 2, 3]],
}
)
table = pa.table(df)
# Arrow -> fastparquet
file_arrow = str(tempdir / "cross_compat_arrow.parquet")
pq.write_table(table, file_arrow, compression=None)
fp_file = fp.ParquetFile(file_arrow)
df_fp = fp_file.to_pandas()
> tm.assert_frame_equal(df, df_fp)
pyarrow/tests/parquet/test_basic.py:741: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ testing.pyx:55: in pandas._libs.testing.assert_almost_equal
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ???
E AssertionError: DataFrame.iloc[:, 3] (column name="d") are different
E E DataFrame.iloc[:, 3] (column name="d") values are different (66.66667 %)
E [index]: [0, 1, 2]
E [left]: [True, False, True]
E [right]: [False, False, False]
E At positional index 0, first diff: True != False
testing.pyx:173: AssertionError

@jorisvandenbosschejorisvandenbosscheSep 25, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Searching for fastparquet in our code base, I assume that we actually don't have any CI build that includes it ... So we are just not running that test anywhere, whoops.

Now, the cross-compatibility is also tested in the pandas test suite, so maybe it's not too important to have it here as well. Anyway, opened an issue at #37853

@jorisvandenbosschejorisvandenbosscheSep 25, 2023•
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although fastparquet reading a column of boolens wrongly for a pyarrow-written file seems a quite serious issue ..
Now, similar data is used in the pandas test suite, where this is passing.

@h-vetinarih-vetinariSep 25, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for raising the issue. I agree it does looks potentially serious, but I hadn't looked too closely because it was just one test. The first thing would be to add testing here in CI. I think this PR is fine as is (I try to keep the test skips to an absolute minimum and remove them whenever possible; that then gets picked up by the next recipe sync anyway)

@jorisvandenbosschejorisvandenbosscheSep 25, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took a closer look at it (could reproduce this locally), and so actually it was good to do so, as this is a problem in fastparquet reading parquet files generated by latest pyarrow, see details at #37853 (comment)

But yes, so skipping for now is perfectly fine.

Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml Outdated
@h-vetinari

h-vetinari commented Sep 17, 2023

Copy link
Copy Markdown
ContributorAuthor

@github-actions crossbow submit -g conda

@github-actions

github-actionsBot commented Sep 17, 2023

Copy link
Copy Markdown

Revision: d7af614

Submitted crossbow builds: ursacomputing/crossbow @ actions-20144be8d4

TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cpu-r43Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cpu-r43Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-arm64-cpu-r43Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r42Azure
conda-osx-x64-cpu-r43Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure

@github-actions

github-actionsBot commented Sep 21, 2023

Copy link
Copy Markdown

Revision: 53a4b70

Submitted crossbow builds: ursacomputing/crossbow @ actions-80a2017be8

TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cpu-r43Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cpu-r43Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-arm64-cpu-r43Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r42Azure
conda-osx-x64-cpu-r43Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure

@h-vetinari

h-vetinari commented Sep 25, 2023

Copy link
Copy Markdown
ContributorAuthor

Failures:

linux-64

Failed during artefact upload -- irrelevant

linux-aarch64

Connection lost to agent -- irrelevant

osx-64

=================================== FAILURES ===================================
_____________ test_debug_memory_pool_disabled[system_memory_pool] ______________
pool_factory = <built-in function system_memory_pool>
@pytest.mark.parametrize('pool_factory', supported_factories())
def test_debug_memory_pool_disabled(pool_factory):
res = run_debug_memory_pool(pool_factory.__name__, "")
# The subprocess either returned successfully or was killed by a signal
# (due to writing out of bounds), depending on the underlying allocator.
if os.name == "posix":
assert res.returncode <= 0
else:
res.check_returncode()
> assert res.stderr == ""
E AssertionError: assert 'libc++abi: P...ion called!\n' == ''
E + libc++abi: Pure virtual function called!
pyarrow/tests/test_memory.py:255: AssertionError
----------------------------- Captured stderr call -----------------------------
libc++abi: Pure virtual function called!

I can skip the respective test on osx if that's what people prefer. Otherwise this PR should be ready.

h-vetinari
h-vetinari commented Sep 25, 2023
View reviewed changes
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml Outdated
@github-actionsgithub-actionsBot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Sep 25, 2023
@github-actionsgithub-actionsBot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Sep 25, 2023
h-vetinari
h-vetinari commented Sep 25, 2023
View reviewed changes
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml Outdated
@jorisvandenbosschejorisvandenbossche mentioned this pull request Sep 25, 2023
Closed
@github-actionsgithub-actionsBot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Sep 25, 2023
@h-vetinari

h-vetinari commented Sep 28, 2023

Copy link
Copy Markdown
ContributorAuthor

Anything else to do here?

@jorisvandenbossche

jorisvandenbossche commented Sep 28, 2023

Copy link
Copy Markdown
Member

I don't think so, thanks for the ping, and for all the work here!

@github-actionsgithub-actionsBot added awaiting merge Awaiting merge and removed awaiting changes Awaiting changes labels Sep 28, 2023
@jorisvandenbossche
jorisvandenbossche merged commit c9674bc into apache:mainSep 28, 2023
@jorisvandenbosschejorisvandenbossche removed the awaiting merge Awaiting merge label Sep 28, 2023
@h-vetinari
h-vetinari deleted the conda_sync branch September 28, 2023 11:58
@conbench-apache-arrow

conbench-apache-arrowBot commented Sep 28, 2023

Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit c9674bc.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.

loicalleyne pushed a commit to loicalleyne/arrow that referenced this pull request Nov 13, 2023
@h-vetinari@loicalleyne
apacheGH-37621: [Packaging][Conda] Sync conda recipes with feedstocks (…
6e5baa3
…apache#37624)
Syncing after the release of 13.0.0 + a couple of migrations (state as of conda-forge/arrow-cpp-feedstock#1168 & conda-forge/r-arrow-feedstock#68)
Relevant updates:
* we're not building twice for different protobuf versions anymore
* new abseil version (fixesapache#36908)
* we've finally upgraded the aws-sdk to 1.11
* the default R versions (on unix) are now 4.2 & 4.3.
Also some further hardening of the activation scripts & clean-ups for dependencies & test skips.
* Closes: apache#37621
Lead-authored-by: H. Vetinari <h.vetinari@gmx.com>
Co-authored-by: h-vetinari <h.vetinari@gmx.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
dgreiss pushed a commit to dgreiss/arrow that referenced this pull request Feb 19, 2024
@h-vetinari@dgreiss
apacheGH-37621: [Packaging][Conda] Sync conda recipes with feedstocks (…
e4438e8
…apache#37624)
Syncing after the release of 13.0.0 + a couple of migrations (state as of conda-forge/arrow-cpp-feedstock#1168 & conda-forge/r-arrow-feedstock#68)
Relevant updates:
* we're not building twice for different protobuf versions anymore
* new abseil version (fixesapache#36908)
* we've finally upgraded the aws-sdk to 1.11
* the default R versions (on unix) are now 4.2 & 4.3.
Also some further hardening of the activation scripts & clean-ups for dependencies & test skips.
* Closes: apache#37621
Lead-authored-by: H. Vetinari <h.vetinari@gmx.com>
Co-authored-by: h-vetinari <h.vetinari@gmx.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
@jorisvandenbosschejorisvandenbossche mentioned this pull request May 17, 2024
Merged
5 tasks
This was referenced Jun 11, 2024
Closed
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@raulcdraulcdraulcd left review comments
@pitroupitroupitrou left review comments
@jorisvandenbosschejorisvandenbosschejorisvandenbossche approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@koukouAwaiting requested review from koukou is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Sync conda-forge recipe [Python][FlightRPC] Tests segfault on OSX in conda-forge

5 participants

@h-vetinari@kou@jorisvandenbossche@raulcd@pitrou
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
GH-37621: [Packaging][Conda] Sync conda recipes with feedstocks - #37624
Merged
jorisvandenbossche merged 9 commits into
apache:mainfrom 
h-vetinari:conda_sync
Sep 28, 2023
Merged
GH-37621: [Packaging][Conda] Sync conda recipes with feedstocks#37624
jorisvandenbossche merged 9 commits into
apache:mainfrom 
h-vetinari:conda_sync
Conversation
@h-vetinari
@h-vetinarih-vetinari
commented
Sep 8, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Syncing after the release of 13.0.0 + a couple of migrations (state as of conda-forge/arrow-cpp-feedstock#1168 & conda-forge/r-arrow-feedstock#68)
Relevant updates:
Also some further hardening of the activation scripts & clean-ups for dependencies & test skips.
@github-actions
github-actionsBot
commented
Sep 8, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #37621has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Sep 8, 2023
@h-vetinari
h-vetinari
commented
Sep 8, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
Sep 8, 2023

Copy link


Copy Markdown

Revision: c04c157
Submitted crossbow builds: ursacomputing/crossbow @ actions-630421cd79
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cpu-r43Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cpu-r43Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-arm64-cpu-r43Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r42Azure
conda-osx-x64-cpu-r43Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting review
Awaiting review
labels
Sep 8, 2023
@h-vetinari
h-vetinari
commented
Sep 8, 2023

Copy link


Copy Markdown

ContributorAuthor
Something is not working out with the CUDA cross-compilation here:
[82/139] Linking CXX shared module release/_cuda.cpython-310-powerpc64le-linux-gnu.so
FAILED: release/_cuda.cpython-310-powerpc64le-linux-gnu.so : && $BUILD_PREFIX/bin/powerpc64le-conda-linux-gnu-c++ -fPIC -Wno-noexcept-type -Wall -fno-semantic-interposition -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -mcpu=power8 -mtune=power8 -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -isystem $PREFIX/include -fdebug-prefix-map=/build/apache-arrow_1694161537567/work=/usr/local/src/conda/pyarrow-14.0.0.dev124 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -isystem /usr/local/cuda/targets/ppc64le-linux/include -fdiagnostics-color=always -fno-omit-frame-pointer -Wno-unused-variable -Wno-maybe-uninitialized -O3 -DNDEBUG -O2 -ftree-vectorize -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--allow-shlib-undefined -Wl,-rpath,$PREFIX/lib -Wl,-rpath-link,$PREFIX/lib -L$PREFIX/lib -shared -o release/_cuda.cpython-310-powerpc64le-linux-gnu.so CMakeFiles/_cuda.dir/_cuda.cpp.o -Wl,-rpath,"\$ORIGIN" release/libarrow_python.so $PREFIX/lib/libarrow_cuda.so.1400.0.0 $PREFIX/lib/libarrow_dataset.so.1400.0.0 $PREFIX/lib/libarrow_acero.so.1400.0.0 $PREFIX/lib/libparquet.so.1400.0.0 $PREFIX/lib/libarrow.so.1400.0.0 /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so && :
$BUILD_PREFIX/powerpc64le-conda-linux-gnu/bin/ld: /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status

It does work in conda-forge, and CUDA on aarch/ppc is a bit more of a niche setup, so I guess we could also drop it here. For completeness, I looked at the diff for the build scripts, but I couldn't determine something that would touch upon where the CUDA_HOME would point -- the error is pretty clearly that we're not pointing to the ppc version of libcuda.so:
$BUILD_PREFIX/powerpc64le-conda-linux-gnu/bin/ld: /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so: error adding symbols: file in wrong format

@h-vetinari
h-vetinari
commented
Sep 8, 2023

Copy link


Copy Markdown

ContributorAuthor
Is it intended that gandiva now has a run-time dependence on libllvm?
 import pyarrow.gandiva
ImportError: libLLVM-15.so: cannot open shared object file: No such file or directory

Adding the respective host-dependence (+ respective run-export) is not hard, I'm just double-checking that this is intentional.
CC @pitrou@kou@raulcd@jorisvandenbossche
PS. This is a rare case where windows looks better than unix, only one test failure:
=========================== short test summary info ===========================
FAILED tests/parquet/test_basic.py::test_fastparquet_cross_compatibility - As...
= 1 failed, 7154 passed, 355 skipped, 12 deselected, 22 xfailed, 2 xpassed, 8 warnings in 302.94s (0:05:02) =

@kou
kou
commented
Sep 8, 2023

Copy link


Copy Markdown

Member
Yes. It's caused by #37412.

We use libLLVM.so by default on non-Windows. If you don't like it you can add -DARROW_LLVM_USE_SHARED=OFF to use static LLVM libraries.

(I think that libLLVM.so is better to reduce duplication.)
@h-vetinarih-vetinari
mentioned this pull request
Sep 8, 2023
 Closed

@h-vetinari
h-vetinari
commented
Sep 12, 2023

Copy link


Copy Markdown

ContributorAuthor
It does work in conda-forge, and CUDA on aarch/ppc is a bit more of a niche setup, so I guess we could also drop it here.
I started debugging this PR on conda-forge infrastructure in conda-forge/arrow-cpp-feedstock#1170, and it turns out that the problem hits us there as well. I'm not sure what changed since 13.0, but it seems something is now overriding (or not respecting) our CUDA_HOME, which is necessary to make cross-compilation work with CUDA...
@kou
kou
commented
Sep 13, 2023

Copy link


Copy Markdown

Member
Could you try -DCUDAToolkit_ROOT=${CUDA_HOME} instead of -DCUDA_TOOLKIT_ROOT_DIR=${CUDA_HOME}?
CUDA_TOOLKIT_ROOT_DIR is for old FindCUDA: https://cmake.org/cmake/help/latest/module/FindCUDA.html

We use FindCUDAToolkit for new CMake: https://cmake.org/cmake/help/latest/module/FindCUDAToolkit.html
@h-vetinari
h-vetinari
commented
Sep 13, 2023

Copy link


Copy Markdown

ContributorAuthor
Thanks for the quick response, that sounds like a very promising candidate! Rebased & retriggered conda-forge/arrow-cpp-feedstock#1170. Will sync back to this PR if passing.
@h-vetinarih-vetinari
mentioned this pull request
Sep 13, 2023

Closed

@github-actionsgithub-actionsBot
added

Component: Python

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Sep 17, 2023
h-vetinari
h-vetinari
commented
Sep 17, 2023
View reviewed changes
Comment threadpython/pyarrow/tests/test_fs.py
Outdated

Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml

Comment on lines

+389
 to 
+390

 # currently broken
 {% set tests_to_skip = tests_to_skip + " or test_fastparquet_cross_compatibility" %}
@h-vetinarih-vetinariSep 17, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't raised an issue for this yet, but this is consistently failing. Not sure what's the difference between our CI and the one here.
@jorisvandenbosschejorisvandenbosscheSep 21, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know a CI build log that shows the error for this? (the last ones will have this already skipped)

(to have an idea what is going on here)
@h-vetinarih-vetinariSep 21, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_____________________ test_fastparquet_cross_compatibility _____________________
tempdir = PosixPath('/tmp/pytest-of-conda/pytest-0/test_fastparquet_cross_compati0')
@pytest.mark.pandas
@pytest.mark.fastparquet
@pytest.mark.filterwarnings("ignore:RangeIndex:FutureWarning")
@pytest.mark.filterwarnings("ignore:tostring:DeprecationWarning:fastparquet")
def test_fastparquet_cross_compatibility(tempdir):
fp = pytest.importorskip('fastparquet')
df = pd.DataFrame(
{
"a": list("abc"),
"b": list(range(1, 4)),
"c": np.arange(4.0, 7.0, dtype="float64"),
"d": [True, False, True],
"e": pd.date_range("20130101", periods=3),
"f": pd.Categorical(["a", "b", "a"]),
# fastparquet writes list as BYTE_ARRAY JSON, so no roundtrip
# "g": [[1, 2], None, [1, 2, 3]],
}
)
table = pa.table(df)
# Arrow -> fastparquet
file_arrow = str(tempdir / "cross_compat_arrow.parquet")
pq.write_table(table, file_arrow, compression=None)
fp_file = fp.ParquetFile(file_arrow)
df_fp = fp_file.to_pandas()
> tm.assert_frame_equal(df, df_fp)
pyarrow/tests/parquet/test_basic.py:741: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ testing.pyx:55: in pandas._libs.testing.assert_almost_equal
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ???
E AssertionError: DataFrame.iloc[:, 3] (column name="d") are different
E E DataFrame.iloc[:, 3] (column name="d") values are different (66.66667 %)
E [index]: [0, 1, 2]
E [left]: [True, False, True]
E [right]: [False, False, False]
E At positional index 0, first diff: True != False
testing.pyx:173: AssertionError

@jorisvandenbosschejorisvandenbosscheSep 25, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Searching for fastparquet in our code base, I assume that we actually don't have any CI build that includes it ... So we are just not running that test anywhere, whoops.
Now, the cross-compatibility is also tested in the pandas test suite, so maybe it's not too important to have it here as well. Anyway, opened an issue at #37853
@jorisvandenbosschejorisvandenbosscheSep 25, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although fastparquet reading a column of boolens wrongly for a pyarrow-written file seems a quite serious issue ..

Now, similar data is used in the pandas test suite, where this is passing.
@h-vetinarih-vetinariSep 25, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for raising the issue. I agree it does looks potentially serious, but I hadn't looked too closely because it was just one test. The first thing would be to add testing here in CI. I think this PR is fine as is (I try to keep the test skips to an absolute minimum and remove them whenever possible; that then gets picked up by the next recipe sync anyway)
@jorisvandenbosschejorisvandenbosscheSep 25, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took a closer look at it (could reproduce this locally), and so actually it was good to do so, as this is a problem in fastparquet reading parquet files generated by latest pyarrow, see details at #37853 (comment)
But yes, so skipping for now is perfectly fine.
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
Outdated

@h-vetinari
h-vetinari
commented
Sep 17, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
Sep 17, 2023

Copy link


Copy Markdown

Revision: d7af614
Submitted crossbow builds: ursacomputing/crossbow @ actions-20144be8d4
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cpu-r43Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cpu-r43Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-arm64-cpu-r43Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r42Azure
conda-osx-x64-cpu-r43Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@github-actions
github-actionsBot
commented
Sep 21, 2023

Copy link


Copy Markdown

Revision: 53a4b70
Submitted crossbow builds: ursacomputing/crossbow @ actions-80a2017be8
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cpu-r43Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cpu-r43Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-arm64-cpu-r43Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r42Azure
conda-osx-x64-cpu-r43Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@h-vetinari
h-vetinari
commented
Sep 25, 2023

Copy link


Copy Markdown

ContributorAuthor
Failures:
linux-64
Failed during artefact upload -- irrelevant
linux-aarch64
Connection lost to agent -- irrelevant
osx-64
=================================== FAILURES ===================================
_____________ test_debug_memory_pool_disabled[system_memory_pool] ______________
pool_factory = <built-in function system_memory_pool>
@pytest.mark.parametrize('pool_factory', supported_factories())
def test_debug_memory_pool_disabled(pool_factory):
res = run_debug_memory_pool(pool_factory.__name__, "")
# The subprocess either returned successfully or was killed by a signal
# (due to writing out of bounds), depending on the underlying allocator.
if os.name == "posix":
assert res.returncode <= 0
else:
res.check_returncode()
> assert res.stderr == ""
E AssertionError: assert 'libc++abi: P...ion called!\n' == ''
E + libc++abi: Pure virtual function called!
pyarrow/tests/test_memory.py:255: AssertionError
----------------------------- Captured stderr call -----------------------------
libc++abi: Pure virtual function called!

I can skip the respective test on osx if that's what people prefer. Otherwise this PR should be ready.
h-vetinari
h-vetinari
commented
Sep 25, 2023
View reviewed changes
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting change review
Awaiting change review
labels
Sep 25, 2023
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Sep 25, 2023
h-vetinari
h-vetinari
commented
Sep 25, 2023
View reviewed changes
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
Outdated

@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Sep 25, 2023

Closed

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting change review
Awaiting change review
labels
Sep 25, 2023
@h-vetinari
h-vetinari
commented
Sep 28, 2023

Copy link


Copy Markdown

ContributorAuthor
Anything else to do here?
@jorisvandenbossche
jorisvandenbossche
commented
Sep 28, 2023

Copy link


Copy Markdown

Member
I don't think so, thanks for the ping, and for all the work here!
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting changes
Awaiting changes
labels
Sep 28, 2023
@jorisvandenbossche
jorisvandenbossche
merged commit c9674bc
into
apache:mainSep 28, 2023
@jorisvandenbosschejorisvandenbossche
removed
the 
awaiting merge
Awaiting merge
label
Sep 28, 2023
@h-vetinari
h-vetinari
deleted the

conda_sync

branch
September 28, 2023 11:58
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Sep 28, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit c9674bc.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@h-vetinari@loicalleyne
apacheGH-37621: [Packaging][Conda] Sync conda recipes with feedstocks (…
6e5baa3
…apache#37624)
Syncing after the release of 13.0.0 + a couple of migrations (state as of conda-forge/arrow-cpp-feedstock#1168 & conda-forge/r-arrow-feedstock#68)
Relevant updates:
* we're not building twice for different protobuf versions anymore
* new abseil version (fixesapache#36908)
* we've finally upgraded the aws-sdk to 1.11
* the default R versions (on unix) are now 4.2 & 4.3.
Also some further hardening of the activation scripts & clean-ups for dependencies & test skips.
* Closes: apache#37621
Lead-authored-by: H. Vetinari <h.vetinari@gmx.com>
Co-authored-by: h-vetinari <h.vetinari@gmx.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
dgreiss pushed a commit
to dgreiss/arrow
that referenced
this pull request
Feb 19, 2024
@h-vetinari@dgreiss
apacheGH-37621: [Packaging][Conda] Sync conda recipes with feedstocks (…
e4438e8
…apache#37624)
Syncing after the release of 13.0.0 + a couple of migrations (state as of conda-forge/arrow-cpp-feedstock#1168 & conda-forge/r-arrow-feedstock#68)
Relevant updates:
* we're not building twice for different protobuf versions anymore
* new abseil version (fixesapache#36908)
* we've finally upgraded the aws-sdk to 1.11
* the default R versions (on unix) are now 4.2 & 4.3.
Also some further hardening of the activation scripts & clean-ups for dependencies & test skips.
* Closes: apache#37621
Lead-authored-by: H. Vetinari <h.vetinari@gmx.com>
Co-authored-by: h-vetinari <h.vetinari@gmx.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
May 17, 2024
 Merged

5 tasks

This was referenced Jun 11, 2024

Closed

 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@raulcdraulcdraulcd left review comments
@pitroupitroupitrou left review comments
@jorisvandenbosschejorisvandenbosschejorisvandenbossche approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@koukouAwaiting requested review from koukou is a code owner

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Sync conda-forge recipe

[Python][FlightRPC] Tests segfault on OSX in conda-forge


5 participants

@h-vetinari@kou@jorisvandenbossche@raulcd@pitrou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-37621: [Packaging][Conda] Sync conda recipes with feedstocks - #37624
Merged
jorisvandenbossche merged 9 commits into
apache:mainfrom 
h-vetinari:conda_sync
Sep 28, 2023
Merged
GH-37621: [Packaging][Conda] Sync conda recipes with feedstocks#37624
jorisvandenbossche merged 9 commits into
apache:mainfrom 
h-vetinari:conda_sync
Conversation
@h-vetinari
@h-vetinarih-vetinari
commented
Sep 8, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Syncing after the release of 13.0.0 + a couple of migrations (state as of conda-forge/arrow-cpp-feedstock#1168 & conda-forge/r-arrow-feedstock#68)
Relevant updates:
Also some further hardening of the activation scripts & clean-ups for dependencies & test skips.
@github-actions
github-actionsBot
commented
Sep 8, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #37621has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Sep 8, 2023
@h-vetinari
h-vetinari
commented
Sep 8, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
Sep 8, 2023

Copy link


Copy Markdown

Revision: c04c157
Submitted crossbow builds: ursacomputing/crossbow @ actions-630421cd79
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cpu-r43Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cpu-r43Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-arm64-cpu-r43Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r42Azure
conda-osx-x64-cpu-r43Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting review
Awaiting review
labels
Sep 8, 2023
@h-vetinari
h-vetinari
commented
Sep 8, 2023

Copy link


Copy Markdown

ContributorAuthor
Something is not working out with the CUDA cross-compilation here:
[82/139] Linking CXX shared module release/_cuda.cpython-310-powerpc64le-linux-gnu.so
FAILED: release/_cuda.cpython-310-powerpc64le-linux-gnu.so : && $BUILD_PREFIX/bin/powerpc64le-conda-linux-gnu-c++ -fPIC -Wno-noexcept-type -Wall -fno-semantic-interposition -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -mcpu=power8 -mtune=power8 -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -isystem $PREFIX/include -fdebug-prefix-map=/build/apache-arrow_1694161537567/work=/usr/local/src/conda/pyarrow-14.0.0.dev124 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -isystem /usr/local/cuda/targets/ppc64le-linux/include -fdiagnostics-color=always -fno-omit-frame-pointer -Wno-unused-variable -Wno-maybe-uninitialized -O3 -DNDEBUG -O2 -ftree-vectorize -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--allow-shlib-undefined -Wl,-rpath,$PREFIX/lib -Wl,-rpath-link,$PREFIX/lib -L$PREFIX/lib -shared -o release/_cuda.cpython-310-powerpc64le-linux-gnu.so CMakeFiles/_cuda.dir/_cuda.cpp.o -Wl,-rpath,"\$ORIGIN" release/libarrow_python.so $PREFIX/lib/libarrow_cuda.so.1400.0.0 $PREFIX/lib/libarrow_dataset.so.1400.0.0 $PREFIX/lib/libarrow_acero.so.1400.0.0 $PREFIX/lib/libparquet.so.1400.0.0 $PREFIX/lib/libarrow.so.1400.0.0 /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so && :
$BUILD_PREFIX/powerpc64le-conda-linux-gnu/bin/ld: /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status

It does work in conda-forge, and CUDA on aarch/ppc is a bit more of a niche setup, so I guess we could also drop it here. For completeness, I looked at the diff for the build scripts, but I couldn't determine something that would touch upon where the CUDA_HOME would point -- the error is pretty clearly that we're not pointing to the ppc version of libcuda.so:
$BUILD_PREFIX/powerpc64le-conda-linux-gnu/bin/ld: /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so: error adding symbols: file in wrong format

@h-vetinari
h-vetinari
commented
Sep 8, 2023

Copy link


Copy Markdown

ContributorAuthor
Is it intended that gandiva now has a run-time dependence on libllvm?
 import pyarrow.gandiva
ImportError: libLLVM-15.so: cannot open shared object file: No such file or directory

Adding the respective host-dependence (+ respective run-export) is not hard, I'm just double-checking that this is intentional.
CC @pitrou@kou@raulcd@jorisvandenbossche
PS. This is a rare case where windows looks better than unix, only one test failure:
=========================== short test summary info ===========================
FAILED tests/parquet/test_basic.py::test_fastparquet_cross_compatibility - As...
= 1 failed, 7154 passed, 355 skipped, 12 deselected, 22 xfailed, 2 xpassed, 8 warnings in 302.94s (0:05:02) =

@kou
kou
commented
Sep 8, 2023

Copy link


Copy Markdown

Member
Yes. It's caused by #37412.

We use libLLVM.so by default on non-Windows. If you don't like it you can add -DARROW_LLVM_USE_SHARED=OFF to use static LLVM libraries.

(I think that libLLVM.so is better to reduce duplication.)
@h-vetinarih-vetinari
mentioned this pull request
Sep 8, 2023
 Closed

@h-vetinari
h-vetinari
commented
Sep 12, 2023

Copy link


Copy Markdown

ContributorAuthor
It does work in conda-forge, and CUDA on aarch/ppc is a bit more of a niche setup, so I guess we could also drop it here.
I started debugging this PR on conda-forge infrastructure in conda-forge/arrow-cpp-feedstock#1170, and it turns out that the problem hits us there as well. I'm not sure what changed since 13.0, but it seems something is now overriding (or not respecting) our CUDA_HOME, which is necessary to make cross-compilation work with CUDA...
@kou
kou
commented
Sep 13, 2023

Copy link


Copy Markdown

Member
Could you try -DCUDAToolkit_ROOT=${CUDA_HOME} instead of -DCUDA_TOOLKIT_ROOT_DIR=${CUDA_HOME}?
CUDA_TOOLKIT_ROOT_DIR is for old FindCUDA: https://cmake.org/cmake/help/latest/module/FindCUDA.html

We use FindCUDAToolkit for new CMake: https://cmake.org/cmake/help/latest/module/FindCUDAToolkit.html
@h-vetinari
h-vetinari
commented
Sep 13, 2023

Copy link


Copy Markdown

ContributorAuthor
Thanks for the quick response, that sounds like a very promising candidate! Rebased & retriggered conda-forge/arrow-cpp-feedstock#1170. Will sync back to this PR if passing.
@h-vetinarih-vetinari
mentioned this pull request
Sep 13, 2023

Closed

@github-actionsgithub-actionsBot
added

Component: Python

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Sep 17, 2023
h-vetinari
h-vetinari
commented
Sep 17, 2023
View reviewed changes
Comment threadpython/pyarrow/tests/test_fs.py
Outdated

Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml

Comment on lines

+389
 to 
+390

 # currently broken
 {% set tests_to_skip = tests_to_skip + " or test_fastparquet_cross_compatibility" %}
@h-vetinarih-vetinariSep 17, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't raised an issue for this yet, but this is consistently failing. Not sure what's the difference between our CI and the one here.
@jorisvandenbosschejorisvandenbosscheSep 21, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know a CI build log that shows the error for this? (the last ones will have this already skipped)

(to have an idea what is going on here)
@h-vetinarih-vetinariSep 21, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_____________________ test_fastparquet_cross_compatibility _____________________
tempdir = PosixPath('/tmp/pytest-of-conda/pytest-0/test_fastparquet_cross_compati0')
@pytest.mark.pandas
@pytest.mark.fastparquet
@pytest.mark.filterwarnings("ignore:RangeIndex:FutureWarning")
@pytest.mark.filterwarnings("ignore:tostring:DeprecationWarning:fastparquet")
def test_fastparquet_cross_compatibility(tempdir):
fp = pytest.importorskip('fastparquet')
df = pd.DataFrame(
{
"a": list("abc"),
"b": list(range(1, 4)),
"c": np.arange(4.0, 7.0, dtype="float64"),
"d": [True, False, True],
"e": pd.date_range("20130101", periods=3),
"f": pd.Categorical(["a", "b", "a"]),
# fastparquet writes list as BYTE_ARRAY JSON, so no roundtrip
# "g": [[1, 2], None, [1, 2, 3]],
}
)
table = pa.table(df)
# Arrow -> fastparquet
file_arrow = str(tempdir / "cross_compat_arrow.parquet")
pq.write_table(table, file_arrow, compression=None)
fp_file = fp.ParquetFile(file_arrow)
df_fp = fp_file.to_pandas()
> tm.assert_frame_equal(df, df_fp)
pyarrow/tests/parquet/test_basic.py:741: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ testing.pyx:55: in pandas._libs.testing.assert_almost_equal
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ???
E AssertionError: DataFrame.iloc[:, 3] (column name="d") are different
E E DataFrame.iloc[:, 3] (column name="d") values are different (66.66667 %)
E [index]: [0, 1, 2]
E [left]: [True, False, True]
E [right]: [False, False, False]
E At positional index 0, first diff: True != False
testing.pyx:173: AssertionError

@jorisvandenbosschejorisvandenbosscheSep 25, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Searching for fastparquet in our code base, I assume that we actually don't have any CI build that includes it ... So we are just not running that test anywhere, whoops.
Now, the cross-compatibility is also tested in the pandas test suite, so maybe it's not too important to have it here as well. Anyway, opened an issue at #37853
@jorisvandenbosschejorisvandenbosscheSep 25, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although fastparquet reading a column of boolens wrongly for a pyarrow-written file seems a quite serious issue ..

Now, similar data is used in the pandas test suite, where this is passing.
@h-vetinarih-vetinariSep 25, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for raising the issue. I agree it does looks potentially serious, but I hadn't looked too closely because it was just one test. The first thing would be to add testing here in CI. I think this PR is fine as is (I try to keep the test skips to an absolute minimum and remove them whenever possible; that then gets picked up by the next recipe sync anyway)
@jorisvandenbosschejorisvandenbosscheSep 25, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took a closer look at it (could reproduce this locally), and so actually it was good to do so, as this is a problem in fastparquet reading parquet files generated by latest pyarrow, see details at #37853 (comment)
But yes, so skipping for now is perfectly fine.
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
Outdated

@h-vetinari
h-vetinari
commented
Sep 17, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
Sep 17, 2023

Copy link


Copy Markdown

Revision: d7af614
Submitted crossbow builds: ursacomputing/crossbow @ actions-20144be8d4
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cpu-r43Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cpu-r43Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-arm64-cpu-r43Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r42Azure
conda-osx-x64-cpu-r43Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@github-actions
github-actionsBot
commented
Sep 21, 2023

Copy link


Copy Markdown

Revision: 53a4b70
Submitted crossbow builds: ursacomputing/crossbow @ actions-80a2017be8
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cpu-r43Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cpu-r43Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-arm64-cpu-r43Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r42Azure
conda-osx-x64-cpu-r43Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@h-vetinari
h-vetinari
commented
Sep 25, 2023

Copy link


Copy Markdown

ContributorAuthor
Failures:
linux-64
Failed during artefact upload -- irrelevant
linux-aarch64
Connection lost to agent -- irrelevant
osx-64
=================================== FAILURES ===================================
_____________ test_debug_memory_pool_disabled[system_memory_pool] ______________
pool_factory = <built-in function system_memory_pool>
@pytest.mark.parametrize('pool_factory', supported_factories())
def test_debug_memory_pool_disabled(pool_factory):
res = run_debug_memory_pool(pool_factory.__name__, "")
# The subprocess either returned successfully or was killed by a signal
# (due to writing out of bounds), depending on the underlying allocator.
if os.name == "posix":
assert res.returncode <= 0
else:
res.check_returncode()
> assert res.stderr == ""
E AssertionError: assert 'libc++abi: P...ion called!\n' == ''
E + libc++abi: Pure virtual function called!
pyarrow/tests/test_memory.py:255: AssertionError
----------------------------- Captured stderr call -----------------------------
libc++abi: Pure virtual function called!

I can skip the respective test on osx if that's what people prefer. Otherwise this PR should be ready.
h-vetinari
h-vetinari
commented
Sep 25, 2023
View reviewed changes
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting change review
Awaiting change review
labels
Sep 25, 2023
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Sep 25, 2023
h-vetinari
h-vetinari
commented
Sep 25, 2023
View reviewed changes
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
Outdated

@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Sep 25, 2023

Closed

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting change review
Awaiting change review
labels
Sep 25, 2023
@h-vetinari
h-vetinari
commented
Sep 28, 2023

Copy link


Copy Markdown

ContributorAuthor
Anything else to do here?
@jorisvandenbossche
jorisvandenbossche
commented
Sep 28, 2023

Copy link


Copy Markdown

Member
I don't think so, thanks for the ping, and for all the work here!
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting changes
Awaiting changes
labels
Sep 28, 2023
@jorisvandenbossche
jorisvandenbossche
merged commit c9674bc
into
apache:mainSep 28, 2023
@jorisvandenbosschejorisvandenbossche
removed
the 
awaiting merge
Awaiting merge
label
Sep 28, 2023
@h-vetinari
h-vetinari
deleted the

conda_sync

branch
September 28, 2023 11:58
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Sep 28, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit c9674bc.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@h-vetinari@loicalleyne
apacheGH-37621: [Packaging][Conda] Sync conda recipes with feedstocks (…
6e5baa3
…apache#37624)
Syncing after the release of 13.0.0 + a couple of migrations (state as of conda-forge/arrow-cpp-feedstock#1168 & conda-forge/r-arrow-feedstock#68)
Relevant updates:
* we're not building twice for different protobuf versions anymore
* new abseil version (fixesapache#36908)
* we've finally upgraded the aws-sdk to 1.11
* the default R versions (on unix) are now 4.2 & 4.3.
Also some further hardening of the activation scripts & clean-ups for dependencies & test skips.
* Closes: apache#37621
Lead-authored-by: H. Vetinari <h.vetinari@gmx.com>
Co-authored-by: h-vetinari <h.vetinari@gmx.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
dgreiss pushed a commit
to dgreiss/arrow
that referenced
this pull request
Feb 19, 2024
@h-vetinari@dgreiss
apacheGH-37621: [Packaging][Conda] Sync conda recipes with feedstocks (…
e4438e8
…apache#37624)
Syncing after the release of 13.0.0 + a couple of migrations (state as of conda-forge/arrow-cpp-feedstock#1168 & conda-forge/r-arrow-feedstock#68)
Relevant updates:
* we're not building twice for different protobuf versions anymore
* new abseil version (fixesapache#36908)
* we've finally upgraded the aws-sdk to 1.11
* the default R versions (on unix) are now 4.2 & 4.3.
Also some further hardening of the activation scripts & clean-ups for dependencies & test skips.
* Closes: apache#37621
Lead-authored-by: H. Vetinari <h.vetinari@gmx.com>
Co-authored-by: h-vetinari <h.vetinari@gmx.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
May 17, 2024
 Merged

5 tasks

This was referenced Jun 11, 2024

Closed

 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@raulcdraulcdraulcd left review comments
@pitroupitroupitrou left review comments
@jorisvandenbosschejorisvandenbosschejorisvandenbossche approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@koukouAwaiting requested review from koukou is a code owner

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Sync conda-forge recipe

[Python][FlightRPC] Tests segfault on OSX in conda-forge


5 participants

@h-vetinari@kou@jorisvandenbossche@raulcd@pitrou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-37621: [Packaging][Conda] Sync conda recipes with feedstocks - #37624
Merged
jorisvandenbossche merged 9 commits into
apache:mainfrom 
h-vetinari:conda_sync
Sep 28, 2023
Merged
GH-37621: [Packaging][Conda] Sync conda recipes with feedstocks#37624
jorisvandenbossche merged 9 commits into
apache:mainfrom 
h-vetinari:conda_sync
Conversation
@h-vetinari
@h-vetinarih-vetinari
commented
Sep 8, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Syncing after the release of 13.0.0 + a couple of migrations (state as of conda-forge/arrow-cpp-feedstock#1168 & conda-forge/r-arrow-feedstock#68)
Relevant updates:
Also some further hardening of the activation scripts & clean-ups for dependencies & test skips.
@github-actions
github-actionsBot
commented
Sep 8, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #37621has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Sep 8, 2023
@h-vetinari
h-vetinari
commented
Sep 8, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
Sep 8, 2023

Copy link


Copy Markdown

Revision: c04c157
Submitted crossbow builds: ursacomputing/crossbow @ actions-630421cd79
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cpu-r43Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cpu-r43Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-arm64-cpu-r43Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r42Azure
conda-osx-x64-cpu-r43Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting review
Awaiting review
labels
Sep 8, 2023
@h-vetinari
h-vetinari
commented
Sep 8, 2023

Copy link


Copy Markdown

ContributorAuthor
Something is not working out with the CUDA cross-compilation here:
[82/139] Linking CXX shared module release/_cuda.cpython-310-powerpc64le-linux-gnu.so
FAILED: release/_cuda.cpython-310-powerpc64le-linux-gnu.so : && $BUILD_PREFIX/bin/powerpc64le-conda-linux-gnu-c++ -fPIC -Wno-noexcept-type -Wall -fno-semantic-interposition -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -mcpu=power8 -mtune=power8 -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -isystem $PREFIX/include -fdebug-prefix-map=/build/apache-arrow_1694161537567/work=/usr/local/src/conda/pyarrow-14.0.0.dev124 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -isystem /usr/local/cuda/targets/ppc64le-linux/include -fdiagnostics-color=always -fno-omit-frame-pointer -Wno-unused-variable -Wno-maybe-uninitialized -O3 -DNDEBUG -O2 -ftree-vectorize -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--allow-shlib-undefined -Wl,-rpath,$PREFIX/lib -Wl,-rpath-link,$PREFIX/lib -L$PREFIX/lib -shared -o release/_cuda.cpython-310-powerpc64le-linux-gnu.so CMakeFiles/_cuda.dir/_cuda.cpp.o -Wl,-rpath,"\$ORIGIN" release/libarrow_python.so $PREFIX/lib/libarrow_cuda.so.1400.0.0 $PREFIX/lib/libarrow_dataset.so.1400.0.0 $PREFIX/lib/libarrow_acero.so.1400.0.0 $PREFIX/lib/libparquet.so.1400.0.0 $PREFIX/lib/libarrow.so.1400.0.0 /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so && :
$BUILD_PREFIX/powerpc64le-conda-linux-gnu/bin/ld: /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status

It does work in conda-forge, and CUDA on aarch/ppc is a bit more of a niche setup, so I guess we could also drop it here. For completeness, I looked at the diff for the build scripts, but I couldn't determine something that would touch upon where the CUDA_HOME would point -- the error is pretty clearly that we're not pointing to the ppc version of libcuda.so:
$BUILD_PREFIX/powerpc64le-conda-linux-gnu/bin/ld: /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so: error adding symbols: file in wrong format

@h-vetinari
h-vetinari
commented
Sep 8, 2023

Copy link


Copy Markdown

ContributorAuthor
Is it intended that gandiva now has a run-time dependence on libllvm?
 import pyarrow.gandiva
ImportError: libLLVM-15.so: cannot open shared object file: No such file or directory

Adding the respective host-dependence (+ respective run-export) is not hard, I'm just double-checking that this is intentional.
CC @pitrou@kou@raulcd@jorisvandenbossche
PS. This is a rare case where windows looks better than unix, only one test failure:
=========================== short test summary info ===========================
FAILED tests/parquet/test_basic.py::test_fastparquet_cross_compatibility - As...
= 1 failed, 7154 passed, 355 skipped, 12 deselected, 22 xfailed, 2 xpassed, 8 warnings in 302.94s (0:05:02) =

@kou
kou
commented
Sep 8, 2023

Copy link


Copy Markdown

Member
Yes. It's caused by #37412.

We use libLLVM.so by default on non-Windows. If you don't like it you can add -DARROW_LLVM_USE_SHARED=OFF to use static LLVM libraries.

(I think that libLLVM.so is better to reduce duplication.)
@h-vetinarih-vetinari
mentioned this pull request
Sep 8, 2023
 Closed

@h-vetinari
h-vetinari
commented
Sep 12, 2023

Copy link


Copy Markdown

ContributorAuthor
It does work in conda-forge, and CUDA on aarch/ppc is a bit more of a niche setup, so I guess we could also drop it here.
I started debugging this PR on conda-forge infrastructure in conda-forge/arrow-cpp-feedstock#1170, and it turns out that the problem hits us there as well. I'm not sure what changed since 13.0, but it seems something is now overriding (or not respecting) our CUDA_HOME, which is necessary to make cross-compilation work with CUDA...
@kou
kou
commented
Sep 13, 2023

Copy link


Copy Markdown

Member
Could you try -DCUDAToolkit_ROOT=${CUDA_HOME} instead of -DCUDA_TOOLKIT_ROOT_DIR=${CUDA_HOME}?
CUDA_TOOLKIT_ROOT_DIR is for old FindCUDA: https://cmake.org/cmake/help/latest/module/FindCUDA.html

We use FindCUDAToolkit for new CMake: https://cmake.org/cmake/help/latest/module/FindCUDAToolkit.html
@h-vetinari
h-vetinari
commented
Sep 13, 2023

Copy link


Copy Markdown

ContributorAuthor
Thanks for the quick response, that sounds like a very promising candidate! Rebased & retriggered conda-forge/arrow-cpp-feedstock#1170. Will sync back to this PR if passing.
@h-vetinarih-vetinari
mentioned this pull request
Sep 13, 2023

Closed

@github-actionsgithub-actionsBot
added

Component: Python

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Sep 17, 2023
h-vetinari
h-vetinari
commented
Sep 17, 2023
View reviewed changes
Comment threadpython/pyarrow/tests/test_fs.py
Outdated

Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml

Comment on lines

+389
 to 
+390

 # currently broken
 {% set tests_to_skip = tests_to_skip + " or test_fastparquet_cross_compatibility" %}
@h-vetinarih-vetinariSep 17, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't raised an issue for this yet, but this is consistently failing. Not sure what's the difference between our CI and the one here.
@jorisvandenbosschejorisvandenbosscheSep 21, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know a CI build log that shows the error for this? (the last ones will have this already skipped)

(to have an idea what is going on here)
@h-vetinarih-vetinariSep 21, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_____________________ test_fastparquet_cross_compatibility _____________________
tempdir = PosixPath('/tmp/pytest-of-conda/pytest-0/test_fastparquet_cross_compati0')
@pytest.mark.pandas
@pytest.mark.fastparquet
@pytest.mark.filterwarnings("ignore:RangeIndex:FutureWarning")
@pytest.mark.filterwarnings("ignore:tostring:DeprecationWarning:fastparquet")
def test_fastparquet_cross_compatibility(tempdir):
fp = pytest.importorskip('fastparquet')
df = pd.DataFrame(
{
"a": list("abc"),
"b": list(range(1, 4)),
"c": np.arange(4.0, 7.0, dtype="float64"),
"d": [True, False, True],
"e": pd.date_range("20130101", periods=3),
"f": pd.Categorical(["a", "b", "a"]),
# fastparquet writes list as BYTE_ARRAY JSON, so no roundtrip
# "g": [[1, 2], None, [1, 2, 3]],
}
)
table = pa.table(df)
# Arrow -> fastparquet
file_arrow = str(tempdir / "cross_compat_arrow.parquet")
pq.write_table(table, file_arrow, compression=None)
fp_file = fp.ParquetFile(file_arrow)
df_fp = fp_file.to_pandas()
> tm.assert_frame_equal(df, df_fp)
pyarrow/tests/parquet/test_basic.py:741: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ testing.pyx:55: in pandas._libs.testing.assert_almost_equal
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ???
E AssertionError: DataFrame.iloc[:, 3] (column name="d") are different
E E DataFrame.iloc[:, 3] (column name="d") values are different (66.66667 %)
E [index]: [0, 1, 2]
E [left]: [True, False, True]
E [right]: [False, False, False]
E At positional index 0, first diff: True != False
testing.pyx:173: AssertionError

@jorisvandenbosschejorisvandenbosscheSep 25, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Searching for fastparquet in our code base, I assume that we actually don't have any CI build that includes it ... So we are just not running that test anywhere, whoops.
Now, the cross-compatibility is also tested in the pandas test suite, so maybe it's not too important to have it here as well. Anyway, opened an issue at #37853
@jorisvandenbosschejorisvandenbosscheSep 25, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although fastparquet reading a column of boolens wrongly for a pyarrow-written file seems a quite serious issue ..

Now, similar data is used in the pandas test suite, where this is passing.
@h-vetinarih-vetinariSep 25, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for raising the issue. I agree it does looks potentially serious, but I hadn't looked too closely because it was just one test. The first thing would be to add testing here in CI. I think this PR is fine as is (I try to keep the test skips to an absolute minimum and remove them whenever possible; that then gets picked up by the next recipe sync anyway)
@jorisvandenbosschejorisvandenbosscheSep 25, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took a closer look at it (could reproduce this locally), and so actually it was good to do so, as this is a problem in fastparquet reading parquet files generated by latest pyarrow, see details at #37853 (comment)
But yes, so skipping for now is perfectly fine.
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
Outdated

@h-vetinari
h-vetinari
commented
Sep 17, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
Sep 17, 2023

Copy link


Copy Markdown

Revision: d7af614
Submitted crossbow builds: ursacomputing/crossbow @ actions-20144be8d4
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cpu-r43Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cpu-r43Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-arm64-cpu-r43Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r42Azure
conda-osx-x64-cpu-r43Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@github-actions
github-actionsBot
commented
Sep 21, 2023

Copy link


Copy Markdown

Revision: 53a4b70
Submitted crossbow builds: ursacomputing/crossbow @ actions-80a2017be8
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cpu-r43Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cpu-r43Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-arm64-cpu-r43Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r42Azure
conda-osx-x64-cpu-r43Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@h-vetinari
h-vetinari
commented
Sep 25, 2023

Copy link


Copy Markdown

ContributorAuthor
Failures:
linux-64
Failed during artefact upload -- irrelevant
linux-aarch64
Connection lost to agent -- irrelevant
osx-64
=================================== FAILURES ===================================
_____________ test_debug_memory_pool_disabled[system_memory_pool] ______________
pool_factory = <built-in function system_memory_pool>
@pytest.mark.parametrize('pool_factory', supported_factories())
def test_debug_memory_pool_disabled(pool_factory):
res = run_debug_memory_pool(pool_factory.__name__, "")
# The subprocess either returned successfully or was killed by a signal
# (due to writing out of bounds), depending on the underlying allocator.
if os.name == "posix":
assert res.returncode <= 0
else:
res.check_returncode()
> assert res.stderr == ""
E AssertionError: assert 'libc++abi: P...ion called!\n' == ''
E + libc++abi: Pure virtual function called!
pyarrow/tests/test_memory.py:255: AssertionError
----------------------------- Captured stderr call -----------------------------
libc++abi: Pure virtual function called!

I can skip the respective test on osx if that's what people prefer. Otherwise this PR should be ready.
h-vetinari
h-vetinari
commented
Sep 25, 2023
View reviewed changes
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting change review
Awaiting change review
labels
Sep 25, 2023
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Sep 25, 2023
h-vetinari
h-vetinari
commented
Sep 25, 2023
View reviewed changes
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
Outdated

@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Sep 25, 2023

Closed

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting change review
Awaiting change review
labels
Sep 25, 2023
@h-vetinari
h-vetinari
commented
Sep 28, 2023

Copy link


Copy Markdown

ContributorAuthor
Anything else to do here?
@jorisvandenbossche
jorisvandenbossche
commented
Sep 28, 2023

Copy link


Copy Markdown

Member
I don't think so, thanks for the ping, and for all the work here!
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting changes
Awaiting changes
labels
Sep 28, 2023
@jorisvandenbossche
jorisvandenbossche
merged commit c9674bc
into
apache:mainSep 28, 2023
@jorisvandenbosschejorisvandenbossche
removed
the 
awaiting merge
Awaiting merge
label
Sep 28, 2023
@h-vetinari
h-vetinari
deleted the

conda_sync

branch
September 28, 2023 11:58
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Sep 28, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit c9674bc.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@h-vetinari@loicalleyne
apacheGH-37621: [Packaging][Conda] Sync conda recipes with feedstocks (…
6e5baa3
…apache#37624)
Syncing after the release of 13.0.0 + a couple of migrations (state as of conda-forge/arrow-cpp-feedstock#1168 & conda-forge/r-arrow-feedstock#68)
Relevant updates:
* we're not building twice for different protobuf versions anymore
* new abseil version (fixesapache#36908)
* we've finally upgraded the aws-sdk to 1.11
* the default R versions (on unix) are now 4.2 & 4.3.
Also some further hardening of the activation scripts & clean-ups for dependencies & test skips.
* Closes: apache#37621
Lead-authored-by: H. Vetinari <h.vetinari@gmx.com>
Co-authored-by: h-vetinari <h.vetinari@gmx.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
dgreiss pushed a commit
to dgreiss/arrow
that referenced
this pull request
Feb 19, 2024
@h-vetinari@dgreiss
apacheGH-37621: [Packaging][Conda] Sync conda recipes with feedstocks (…
e4438e8
…apache#37624)
Syncing after the release of 13.0.0 + a couple of migrations (state as of conda-forge/arrow-cpp-feedstock#1168 & conda-forge/r-arrow-feedstock#68)
Relevant updates:
* we're not building twice for different protobuf versions anymore
* new abseil version (fixesapache#36908)
* we've finally upgraded the aws-sdk to 1.11
* the default R versions (on unix) are now 4.2 & 4.3.
Also some further hardening of the activation scripts & clean-ups for dependencies & test skips.
* Closes: apache#37621
Lead-authored-by: H. Vetinari <h.vetinari@gmx.com>
Co-authored-by: h-vetinari <h.vetinari@gmx.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
May 17, 2024
 Merged

5 tasks

This was referenced Jun 11, 2024

Closed

 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@raulcdraulcdraulcd left review comments
@pitroupitroupitrou left review comments
@jorisvandenbosschejorisvandenbosschejorisvandenbossche approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@koukouAwaiting requested review from koukou is a code owner

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Sync conda-forge recipe

[Python][FlightRPC] Tests segfault on OSX in conda-forge


5 participants

@h-vetinari@kou@jorisvandenbossche@raulcd@pitrou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
GH-37621: [Packaging][Conda] Sync conda recipes with feedstocks - #37624
Merged
jorisvandenbossche merged 9 commits into
apache:mainfrom 
h-vetinari:conda_sync
Sep 28, 2023
Merged
GH-37621: [Packaging][Conda] Sync conda recipes with feedstocks#37624
jorisvandenbossche merged 9 commits into
apache:mainfrom 
h-vetinari:conda_sync
Conversation
@h-vetinari
@h-vetinarih-vetinari
commented
Sep 8, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Syncing after the release of 13.0.0 + a couple of migrations (state as of conda-forge/arrow-cpp-feedstock#1168 & conda-forge/r-arrow-feedstock#68)
Relevant updates:
Also some further hardening of the activation scripts & clean-ups for dependencies & test skips.
@github-actions
github-actionsBot
commented
Sep 8, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #37621has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Sep 8, 2023
@h-vetinari
h-vetinari
commented
Sep 8, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
Sep 8, 2023

Copy link


Copy Markdown

Revision: c04c157
Submitted crossbow builds: ursacomputing/crossbow @ actions-630421cd79
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cpu-r43Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cpu-r43Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-arm64-cpu-r43Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r42Azure
conda-osx-x64-cpu-r43Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting review
Awaiting review
labels
Sep 8, 2023
@h-vetinari
h-vetinari
commented
Sep 8, 2023

Copy link


Copy Markdown

ContributorAuthor
Something is not working out with the CUDA cross-compilation here:
[82/139] Linking CXX shared module release/_cuda.cpython-310-powerpc64le-linux-gnu.so
FAILED: release/_cuda.cpython-310-powerpc64le-linux-gnu.so : && $BUILD_PREFIX/bin/powerpc64le-conda-linux-gnu-c++ -fPIC -Wno-noexcept-type -Wall -fno-semantic-interposition -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -mcpu=power8 -mtune=power8 -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -isystem $PREFIX/include -fdebug-prefix-map=/build/apache-arrow_1694161537567/work=/usr/local/src/conda/pyarrow-14.0.0.dev124 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -isystem /usr/local/cuda/targets/ppc64le-linux/include -fdiagnostics-color=always -fno-omit-frame-pointer -Wno-unused-variable -Wno-maybe-uninitialized -O3 -DNDEBUG -O2 -ftree-vectorize -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--allow-shlib-undefined -Wl,-rpath,$PREFIX/lib -Wl,-rpath-link,$PREFIX/lib -L$PREFIX/lib -shared -o release/_cuda.cpython-310-powerpc64le-linux-gnu.so CMakeFiles/_cuda.dir/_cuda.cpp.o -Wl,-rpath,"\$ORIGIN" release/libarrow_python.so $PREFIX/lib/libarrow_cuda.so.1400.0.0 $PREFIX/lib/libarrow_dataset.so.1400.0.0 $PREFIX/lib/libarrow_acero.so.1400.0.0 $PREFIX/lib/libparquet.so.1400.0.0 $PREFIX/lib/libarrow.so.1400.0.0 /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so && :
$BUILD_PREFIX/powerpc64le-conda-linux-gnu/bin/ld: /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status

It does work in conda-forge, and CUDA on aarch/ppc is a bit more of a niche setup, so I guess we could also drop it here. For completeness, I looked at the diff for the build scripts, but I couldn't determine something that would touch upon where the CUDA_HOME would point -- the error is pretty clearly that we're not pointing to the ppc version of libcuda.so:
$BUILD_PREFIX/powerpc64le-conda-linux-gnu/bin/ld: /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so: error adding symbols: file in wrong format

@h-vetinari
h-vetinari
commented
Sep 8, 2023

Copy link


Copy Markdown

ContributorAuthor
Is it intended that gandiva now has a run-time dependence on libllvm?
 import pyarrow.gandiva
ImportError: libLLVM-15.so: cannot open shared object file: No such file or directory

Adding the respective host-dependence (+ respective run-export) is not hard, I'm just double-checking that this is intentional.
CC @pitrou@kou@raulcd@jorisvandenbossche
PS. This is a rare case where windows looks better than unix, only one test failure:
=========================== short test summary info ===========================
FAILED tests/parquet/test_basic.py::test_fastparquet_cross_compatibility - As...
= 1 failed, 7154 passed, 355 skipped, 12 deselected, 22 xfailed, 2 xpassed, 8 warnings in 302.94s (0:05:02) =

@kou
kou
commented
Sep 8, 2023

Copy link


Copy Markdown

Member
Yes. It's caused by #37412.

We use libLLVM.so by default on non-Windows. If you don't like it you can add -DARROW_LLVM_USE_SHARED=OFF to use static LLVM libraries.

(I think that libLLVM.so is better to reduce duplication.)
@h-vetinarih-vetinari
mentioned this pull request
Sep 8, 2023
 Closed

@h-vetinari
h-vetinari
commented
Sep 12, 2023

Copy link


Copy Markdown

ContributorAuthor
It does work in conda-forge, and CUDA on aarch/ppc is a bit more of a niche setup, so I guess we could also drop it here.
I started debugging this PR on conda-forge infrastructure in conda-forge/arrow-cpp-feedstock#1170, and it turns out that the problem hits us there as well. I'm not sure what changed since 13.0, but it seems something is now overriding (or not respecting) our CUDA_HOME, which is necessary to make cross-compilation work with CUDA...
@kou
kou
commented
Sep 13, 2023

Copy link


Copy Markdown

Member
Could you try -DCUDAToolkit_ROOT=${CUDA_HOME} instead of -DCUDA_TOOLKIT_ROOT_DIR=${CUDA_HOME}?
CUDA_TOOLKIT_ROOT_DIR is for old FindCUDA: https://cmake.org/cmake/help/latest/module/FindCUDA.html

We use FindCUDAToolkit for new CMake: https://cmake.org/cmake/help/latest/module/FindCUDAToolkit.html
@h-vetinari
h-vetinari
commented
Sep 13, 2023

Copy link


Copy Markdown

ContributorAuthor
Thanks for the quick response, that sounds like a very promising candidate! Rebased & retriggered conda-forge/arrow-cpp-feedstock#1170. Will sync back to this PR if passing.
@h-vetinarih-vetinari
mentioned this pull request
Sep 13, 2023

Closed

@github-actionsgithub-actionsBot
added

Component: Python

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Sep 17, 2023
h-vetinari
h-vetinari
commented
Sep 17, 2023
View reviewed changes
Comment threadpython/pyarrow/tests/test_fs.py
Outdated

Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml

Comment on lines

+389
 to 
+390

 # currently broken
 {% set tests_to_skip = tests_to_skip + " or test_fastparquet_cross_compatibility" %}
@h-vetinarih-vetinariSep 17, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't raised an issue for this yet, but this is consistently failing. Not sure what's the difference between our CI and the one here.
@jorisvandenbosschejorisvandenbosscheSep 21, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know a CI build log that shows the error for this? (the last ones will have this already skipped)

(to have an idea what is going on here)
@h-vetinarih-vetinariSep 21, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_____________________ test_fastparquet_cross_compatibility _____________________
tempdir = PosixPath('/tmp/pytest-of-conda/pytest-0/test_fastparquet_cross_compati0')
@pytest.mark.pandas
@pytest.mark.fastparquet
@pytest.mark.filterwarnings("ignore:RangeIndex:FutureWarning")
@pytest.mark.filterwarnings("ignore:tostring:DeprecationWarning:fastparquet")
def test_fastparquet_cross_compatibility(tempdir):
fp = pytest.importorskip('fastparquet')
df = pd.DataFrame(
{
"a": list("abc"),
"b": list(range(1, 4)),
"c": np.arange(4.0, 7.0, dtype="float64"),
"d": [True, False, True],
"e": pd.date_range("20130101", periods=3),
"f": pd.Categorical(["a", "b", "a"]),
# fastparquet writes list as BYTE_ARRAY JSON, so no roundtrip
# "g": [[1, 2], None, [1, 2, 3]],
}
)
table = pa.table(df)
# Arrow -> fastparquet
file_arrow = str(tempdir / "cross_compat_arrow.parquet")
pq.write_table(table, file_arrow, compression=None)
fp_file = fp.ParquetFile(file_arrow)
df_fp = fp_file.to_pandas()
> tm.assert_frame_equal(df, df_fp)
pyarrow/tests/parquet/test_basic.py:741: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ testing.pyx:55: in pandas._libs.testing.assert_almost_equal
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ???
E AssertionError: DataFrame.iloc[:, 3] (column name="d") are different
E E DataFrame.iloc[:, 3] (column name="d") values are different (66.66667 %)
E [index]: [0, 1, 2]
E [left]: [True, False, True]
E [right]: [False, False, False]
E At positional index 0, first diff: True != False
testing.pyx:173: AssertionError

@jorisvandenbosschejorisvandenbosscheSep 25, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Searching for fastparquet in our code base, I assume that we actually don't have any CI build that includes it ... So we are just not running that test anywhere, whoops.
Now, the cross-compatibility is also tested in the pandas test suite, so maybe it's not too important to have it here as well. Anyway, opened an issue at #37853
@jorisvandenbosschejorisvandenbosscheSep 25, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although fastparquet reading a column of boolens wrongly for a pyarrow-written file seems a quite serious issue ..

Now, similar data is used in the pandas test suite, where this is passing.
@h-vetinarih-vetinariSep 25, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for raising the issue. I agree it does looks potentially serious, but I hadn't looked too closely because it was just one test. The first thing would be to add testing here in CI. I think this PR is fine as is (I try to keep the test skips to an absolute minimum and remove them whenever possible; that then gets picked up by the next recipe sync anyway)
@jorisvandenbosschejorisvandenbosscheSep 25, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took a closer look at it (could reproduce this locally), and so actually it was good to do so, as this is a problem in fastparquet reading parquet files generated by latest pyarrow, see details at #37853 (comment)
But yes, so skipping for now is perfectly fine.
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
Outdated

@h-vetinari
h-vetinari
commented
Sep 17, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
Sep 17, 2023

Copy link


Copy Markdown

Revision: d7af614
Submitted crossbow builds: ursacomputing/crossbow @ actions-20144be8d4
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cpu-r43Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cpu-r43Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-arm64-cpu-r43Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r42Azure
conda-osx-x64-cpu-r43Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@github-actions
github-actionsBot
commented
Sep 21, 2023

Copy link


Copy Markdown

Revision: 53a4b70
Submitted crossbow builds: ursacomputing/crossbow @ actions-80a2017be8
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cpu-r43Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cpu-r43Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-arm64-cpu-r43Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r42Azure
conda-osx-x64-cpu-r43Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@h-vetinari
h-vetinari
commented
Sep 25, 2023

Copy link


Copy Markdown

ContributorAuthor
Failures:
linux-64
Failed during artefact upload -- irrelevant
linux-aarch64
Connection lost to agent -- irrelevant
osx-64
=================================== FAILURES ===================================
_____________ test_debug_memory_pool_disabled[system_memory_pool] ______________
pool_factory = <built-in function system_memory_pool>
@pytest.mark.parametrize('pool_factory', supported_factories())
def test_debug_memory_pool_disabled(pool_factory):
res = run_debug_memory_pool(pool_factory.__name__, "")
# The subprocess either returned successfully or was killed by a signal
# (due to writing out of bounds), depending on the underlying allocator.
if os.name == "posix":
assert res.returncode <= 0
else:
res.check_returncode()
> assert res.stderr == ""
E AssertionError: assert 'libc++abi: P...ion called!\n' == ''
E + libc++abi: Pure virtual function called!
pyarrow/tests/test_memory.py:255: AssertionError
----------------------------- Captured stderr call -----------------------------
libc++abi: Pure virtual function called!

I can skip the respective test on osx if that's what people prefer. Otherwise this PR should be ready.
h-vetinari
h-vetinari
commented
Sep 25, 2023
View reviewed changes
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting change review
Awaiting change review
labels
Sep 25, 2023
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Sep 25, 2023
h-vetinari
h-vetinari
commented
Sep 25, 2023
View reviewed changes
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
Outdated

@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Sep 25, 2023

Closed

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting change review
Awaiting change review
labels
Sep 25, 2023
@h-vetinari
h-vetinari
commented
Sep 28, 2023

Copy link


Copy Markdown

ContributorAuthor
Anything else to do here?
@jorisvandenbossche
jorisvandenbossche
commented
Sep 28, 2023

Copy link


Copy Markdown

Member
I don't think so, thanks for the ping, and for all the work here!
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting changes
Awaiting changes
labels
Sep 28, 2023
@jorisvandenbossche
jorisvandenbossche
merged commit c9674bc
into
apache:mainSep 28, 2023
@jorisvandenbosschejorisvandenbossche
removed
the 
awaiting merge
Awaiting merge
label
Sep 28, 2023
@h-vetinari
h-vetinari
deleted the

conda_sync

branch
September 28, 2023 11:58
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Sep 28, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit c9674bc.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@h-vetinari@loicalleyne
apacheGH-37621: [Packaging][Conda] Sync conda recipes with feedstocks (…
6e5baa3
…apache#37624)
Syncing after the release of 13.0.0 + a couple of migrations (state as of conda-forge/arrow-cpp-feedstock#1168 & conda-forge/r-arrow-feedstock#68)
Relevant updates:
* we're not building twice for different protobuf versions anymore
* new abseil version (fixesapache#36908)
* we've finally upgraded the aws-sdk to 1.11
* the default R versions (on unix) are now 4.2 & 4.3.
Also some further hardening of the activation scripts & clean-ups for dependencies & test skips.
* Closes: apache#37621
Lead-authored-by: H. Vetinari <h.vetinari@gmx.com>
Co-authored-by: h-vetinari <h.vetinari@gmx.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
dgreiss pushed a commit
to dgreiss/arrow
that referenced
this pull request
Feb 19, 2024
@h-vetinari@dgreiss
apacheGH-37621: [Packaging][Conda] Sync conda recipes with feedstocks (…
e4438e8
…apache#37624)
Syncing after the release of 13.0.0 + a couple of migrations (state as of conda-forge/arrow-cpp-feedstock#1168 & conda-forge/r-arrow-feedstock#68)
Relevant updates:
* we're not building twice for different protobuf versions anymore
* new abseil version (fixesapache#36908)
* we've finally upgraded the aws-sdk to 1.11
* the default R versions (on unix) are now 4.2 & 4.3.
Also some further hardening of the activation scripts & clean-ups for dependencies & test skips.
* Closes: apache#37621
Lead-authored-by: H. Vetinari <h.vetinari@gmx.com>
Co-authored-by: h-vetinari <h.vetinari@gmx.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
May 17, 2024
 Merged

5 tasks

This was referenced Jun 11, 2024

Closed

 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@raulcdraulcdraulcd left review comments
@pitroupitroupitrou left review comments
@jorisvandenbosschejorisvandenbosschejorisvandenbossche approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@koukouAwaiting requested review from koukou is a code owner

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Sync conda-forge recipe

[Python][FlightRPC] Tests segfault on OSX in conda-forge


5 participants

@h-vetinari@kou@jorisvandenbossche@raulcd@pitrou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-37621: [Packaging][Conda] Sync conda recipes with feedstocks - #37624
Merged
jorisvandenbossche merged 9 commits into
apache:mainfrom 
h-vetinari:conda_sync
Sep 28, 2023
Merged
GH-37621: [Packaging][Conda] Sync conda recipes with feedstocks#37624
jorisvandenbossche merged 9 commits into
apache:mainfrom 
h-vetinari:conda_sync
Conversation
@h-vetinari
@h-vetinarih-vetinari
commented
Sep 8, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Syncing after the release of 13.0.0 + a couple of migrations (state as of conda-forge/arrow-cpp-feedstock#1168 & conda-forge/r-arrow-feedstock#68)
Relevant updates:
Also some further hardening of the activation scripts & clean-ups for dependencies & test skips.
@github-actions
github-actionsBot
commented
Sep 8, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #37621has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Sep 8, 2023
@h-vetinari
h-vetinari
commented
Sep 8, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
Sep 8, 2023

Copy link


Copy Markdown

Revision: c04c157
Submitted crossbow builds: ursacomputing/crossbow @ actions-630421cd79
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cpu-r43Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cpu-r43Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-arm64-cpu-r43Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r42Azure
conda-osx-x64-cpu-r43Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting review
Awaiting review
labels
Sep 8, 2023
@h-vetinari
h-vetinari
commented
Sep 8, 2023

Copy link


Copy Markdown

ContributorAuthor
Something is not working out with the CUDA cross-compilation here:
[82/139] Linking CXX shared module release/_cuda.cpython-310-powerpc64le-linux-gnu.so
FAILED: release/_cuda.cpython-310-powerpc64le-linux-gnu.so : && $BUILD_PREFIX/bin/powerpc64le-conda-linux-gnu-c++ -fPIC -Wno-noexcept-type -Wall -fno-semantic-interposition -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -mcpu=power8 -mtune=power8 -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -isystem $PREFIX/include -fdebug-prefix-map=/build/apache-arrow_1694161537567/work=/usr/local/src/conda/pyarrow-14.0.0.dev124 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -isystem /usr/local/cuda/targets/ppc64le-linux/include -fdiagnostics-color=always -fno-omit-frame-pointer -Wno-unused-variable -Wno-maybe-uninitialized -O3 -DNDEBUG -O2 -ftree-vectorize -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--allow-shlib-undefined -Wl,-rpath,$PREFIX/lib -Wl,-rpath-link,$PREFIX/lib -L$PREFIX/lib -shared -o release/_cuda.cpython-310-powerpc64le-linux-gnu.so CMakeFiles/_cuda.dir/_cuda.cpp.o -Wl,-rpath,"\$ORIGIN" release/libarrow_python.so $PREFIX/lib/libarrow_cuda.so.1400.0.0 $PREFIX/lib/libarrow_dataset.so.1400.0.0 $PREFIX/lib/libarrow_acero.so.1400.0.0 $PREFIX/lib/libparquet.so.1400.0.0 $PREFIX/lib/libarrow.so.1400.0.0 /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so && :
$BUILD_PREFIX/powerpc64le-conda-linux-gnu/bin/ld: /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status

It does work in conda-forge, and CUDA on aarch/ppc is a bit more of a niche setup, so I guess we could also drop it here. For completeness, I looked at the diff for the build scripts, but I couldn't determine something that would touch upon where the CUDA_HOME would point -- the error is pretty clearly that we're not pointing to the ppc version of libcuda.so:
$BUILD_PREFIX/powerpc64le-conda-linux-gnu/bin/ld: /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so: error adding symbols: file in wrong format

@h-vetinari
h-vetinari
commented
Sep 8, 2023

Copy link


Copy Markdown

ContributorAuthor
Is it intended that gandiva now has a run-time dependence on libllvm?
 import pyarrow.gandiva
ImportError: libLLVM-15.so: cannot open shared object file: No such file or directory

Adding the respective host-dependence (+ respective run-export) is not hard, I'm just double-checking that this is intentional.
CC @pitrou@kou@raulcd@jorisvandenbossche
PS. This is a rare case where windows looks better than unix, only one test failure:
=========================== short test summary info ===========================
FAILED tests/parquet/test_basic.py::test_fastparquet_cross_compatibility - As...
= 1 failed, 7154 passed, 355 skipped, 12 deselected, 22 xfailed, 2 xpassed, 8 warnings in 302.94s (0:05:02) =

@kou
kou
commented
Sep 8, 2023

Copy link


Copy Markdown

Member
Yes. It's caused by #37412.

We use libLLVM.so by default on non-Windows. If you don't like it you can add -DARROW_LLVM_USE_SHARED=OFF to use static LLVM libraries.

(I think that libLLVM.so is better to reduce duplication.)
@h-vetinarih-vetinari
mentioned this pull request
Sep 8, 2023
 Closed

@h-vetinari
h-vetinari
commented
Sep 12, 2023

Copy link


Copy Markdown

ContributorAuthor
It does work in conda-forge, and CUDA on aarch/ppc is a bit more of a niche setup, so I guess we could also drop it here.
I started debugging this PR on conda-forge infrastructure in conda-forge/arrow-cpp-feedstock#1170, and it turns out that the problem hits us there as well. I'm not sure what changed since 13.0, but it seems something is now overriding (or not respecting) our CUDA_HOME, which is necessary to make cross-compilation work with CUDA...
@kou
kou
commented
Sep 13, 2023

Copy link


Copy Markdown

Member
Could you try -DCUDAToolkit_ROOT=${CUDA_HOME} instead of -DCUDA_TOOLKIT_ROOT_DIR=${CUDA_HOME}?
CUDA_TOOLKIT_ROOT_DIR is for old FindCUDA: https://cmake.org/cmake/help/latest/module/FindCUDA.html

We use FindCUDAToolkit for new CMake: https://cmake.org/cmake/help/latest/module/FindCUDAToolkit.html
@h-vetinari
h-vetinari
commented
Sep 13, 2023

Copy link


Copy Markdown

ContributorAuthor
Thanks for the quick response, that sounds like a very promising candidate! Rebased & retriggered conda-forge/arrow-cpp-feedstock#1170. Will sync back to this PR if passing.
@h-vetinarih-vetinari
mentioned this pull request
Sep 13, 2023

Closed

@github-actionsgithub-actionsBot
added

Component: Python

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Sep 17, 2023
h-vetinari
h-vetinari
commented
Sep 17, 2023
View reviewed changes
Comment threadpython/pyarrow/tests/test_fs.py
Outdated

Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml

Comment on lines

+389
 to 
+390

 # currently broken
 {% set tests_to_skip = tests_to_skip + " or test_fastparquet_cross_compatibility" %}
@h-vetinarih-vetinariSep 17, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't raised an issue for this yet, but this is consistently failing. Not sure what's the difference between our CI and the one here.
@jorisvandenbosschejorisvandenbosscheSep 21, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know a CI build log that shows the error for this? (the last ones will have this already skipped)

(to have an idea what is going on here)
@h-vetinarih-vetinariSep 21, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_____________________ test_fastparquet_cross_compatibility _____________________
tempdir = PosixPath('/tmp/pytest-of-conda/pytest-0/test_fastparquet_cross_compati0')
@pytest.mark.pandas
@pytest.mark.fastparquet
@pytest.mark.filterwarnings("ignore:RangeIndex:FutureWarning")
@pytest.mark.filterwarnings("ignore:tostring:DeprecationWarning:fastparquet")
def test_fastparquet_cross_compatibility(tempdir):
fp = pytest.importorskip('fastparquet')
df = pd.DataFrame(
{
"a": list("abc"),
"b": list(range(1, 4)),
"c": np.arange(4.0, 7.0, dtype="float64"),
"d": [True, False, True],
"e": pd.date_range("20130101", periods=3),
"f": pd.Categorical(["a", "b", "a"]),
# fastparquet writes list as BYTE_ARRAY JSON, so no roundtrip
# "g": [[1, 2], None, [1, 2, 3]],
}
)
table = pa.table(df)
# Arrow -> fastparquet
file_arrow = str(tempdir / "cross_compat_arrow.parquet")
pq.write_table(table, file_arrow, compression=None)
fp_file = fp.ParquetFile(file_arrow)
df_fp = fp_file.to_pandas()
> tm.assert_frame_equal(df, df_fp)
pyarrow/tests/parquet/test_basic.py:741: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ testing.pyx:55: in pandas._libs.testing.assert_almost_equal
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ???
E AssertionError: DataFrame.iloc[:, 3] (column name="d") are different
E E DataFrame.iloc[:, 3] (column name="d") values are different (66.66667 %)
E [index]: [0, 1, 2]
E [left]: [True, False, True]
E [right]: [False, False, False]
E At positional index 0, first diff: True != False
testing.pyx:173: AssertionError

@jorisvandenbosschejorisvandenbosscheSep 25, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Searching for fastparquet in our code base, I assume that we actually don't have any CI build that includes it ... So we are just not running that test anywhere, whoops.
Now, the cross-compatibility is also tested in the pandas test suite, so maybe it's not too important to have it here as well. Anyway, opened an issue at #37853
@jorisvandenbosschejorisvandenbosscheSep 25, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although fastparquet reading a column of boolens wrongly for a pyarrow-written file seems a quite serious issue ..

Now, similar data is used in the pandas test suite, where this is passing.
@h-vetinarih-vetinariSep 25, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for raising the issue. I agree it does looks potentially serious, but I hadn't looked too closely because it was just one test. The first thing would be to add testing here in CI. I think this PR is fine as is (I try to keep the test skips to an absolute minimum and remove them whenever possible; that then gets picked up by the next recipe sync anyway)
@jorisvandenbosschejorisvandenbosscheSep 25, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took a closer look at it (could reproduce this locally), and so actually it was good to do so, as this is a problem in fastparquet reading parquet files generated by latest pyarrow, see details at #37853 (comment)
But yes, so skipping for now is perfectly fine.
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
Outdated

@h-vetinari
h-vetinari
commented
Sep 17, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
Sep 17, 2023

Copy link


Copy Markdown

Revision: d7af614
Submitted crossbow builds: ursacomputing/crossbow @ actions-20144be8d4
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cpu-r43Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cpu-r43Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-arm64-cpu-r43Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r42Azure
conda-osx-x64-cpu-r43Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@github-actions
github-actionsBot
commented
Sep 21, 2023

Copy link


Copy Markdown

Revision: 53a4b70
Submitted crossbow builds: ursacomputing/crossbow @ actions-80a2017be8
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cpu-r43Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cpu-r43Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-arm64-cpu-r43Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r42Azure
conda-osx-x64-cpu-r43Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@h-vetinari
h-vetinari
commented
Sep 25, 2023

Copy link


Copy Markdown

ContributorAuthor
Failures:
linux-64
Failed during artefact upload -- irrelevant
linux-aarch64
Connection lost to agent -- irrelevant
osx-64
=================================== FAILURES ===================================
_____________ test_debug_memory_pool_disabled[system_memory_pool] ______________
pool_factory = <built-in function system_memory_pool>
@pytest.mark.parametrize('pool_factory', supported_factories())
def test_debug_memory_pool_disabled(pool_factory):
res = run_debug_memory_pool(pool_factory.__name__, "")
# The subprocess either returned successfully or was killed by a signal
# (due to writing out of bounds), depending on the underlying allocator.
if os.name == "posix":
assert res.returncode <= 0
else:
res.check_returncode()
> assert res.stderr == ""
E AssertionError: assert 'libc++abi: P...ion called!\n' == ''
E + libc++abi: Pure virtual function called!
pyarrow/tests/test_memory.py:255: AssertionError
----------------------------- Captured stderr call -----------------------------
libc++abi: Pure virtual function called!

I can skip the respective test on osx if that's what people prefer. Otherwise this PR should be ready.
h-vetinari
h-vetinari
commented
Sep 25, 2023
View reviewed changes
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting change review
Awaiting change review
labels
Sep 25, 2023
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Sep 25, 2023
h-vetinari
h-vetinari
commented
Sep 25, 2023
View reviewed changes
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
Outdated

@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Sep 25, 2023

Closed

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting change review
Awaiting change review
labels
Sep 25, 2023
@h-vetinari
h-vetinari
commented
Sep 28, 2023

Copy link


Copy Markdown

ContributorAuthor
Anything else to do here?
@jorisvandenbossche
jorisvandenbossche
commented
Sep 28, 2023

Copy link


Copy Markdown

Member
I don't think so, thanks for the ping, and for all the work here!
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting changes
Awaiting changes
labels
Sep 28, 2023
@jorisvandenbossche
jorisvandenbossche
merged commit c9674bc
into
apache:mainSep 28, 2023
@jorisvandenbosschejorisvandenbossche
removed
the 
awaiting merge
Awaiting merge
label
Sep 28, 2023
@h-vetinari
h-vetinari
deleted the

conda_sync

branch
September 28, 2023 11:58
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Sep 28, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit c9674bc.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@h-vetinari@loicalleyne
apacheGH-37621: [Packaging][Conda] Sync conda recipes with feedstocks (…
6e5baa3
…apache#37624)
Syncing after the release of 13.0.0 + a couple of migrations (state as of conda-forge/arrow-cpp-feedstock#1168 & conda-forge/r-arrow-feedstock#68)
Relevant updates:
* we're not building twice for different protobuf versions anymore
* new abseil version (fixesapache#36908)
* we've finally upgraded the aws-sdk to 1.11
* the default R versions (on unix) are now 4.2 & 4.3.
Also some further hardening of the activation scripts & clean-ups for dependencies & test skips.
* Closes: apache#37621
Lead-authored-by: H. Vetinari <h.vetinari@gmx.com>
Co-authored-by: h-vetinari <h.vetinari@gmx.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
dgreiss pushed a commit
to dgreiss/arrow
that referenced
this pull request
Feb 19, 2024
@h-vetinari@dgreiss
apacheGH-37621: [Packaging][Conda] Sync conda recipes with feedstocks (…
e4438e8
…apache#37624)
Syncing after the release of 13.0.0 + a couple of migrations (state as of conda-forge/arrow-cpp-feedstock#1168 & conda-forge/r-arrow-feedstock#68)
Relevant updates:
* we're not building twice for different protobuf versions anymore
* new abseil version (fixesapache#36908)
* we've finally upgraded the aws-sdk to 1.11
* the default R versions (on unix) are now 4.2 & 4.3.
Also some further hardening of the activation scripts & clean-ups for dependencies & test skips.
* Closes: apache#37621
Lead-authored-by: H. Vetinari <h.vetinari@gmx.com>
Co-authored-by: h-vetinari <h.vetinari@gmx.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
May 17, 2024
 Merged

5 tasks

This was referenced Jun 11, 2024

Closed

 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@raulcdraulcdraulcd left review comments
@pitroupitroupitrou left review comments
@jorisvandenbosschejorisvandenbosschejorisvandenbossche approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@koukouAwaiting requested review from koukou is a code owner

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Sync conda-forge recipe

[Python][FlightRPC] Tests segfault on OSX in conda-forge


5 participants

@h-vetinari@kou@jorisvandenbossche@raulcd@pitrou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-37621: [Packaging][Conda] Sync conda recipes with feedstocks - #37624
Merged
jorisvandenbossche merged 9 commits into
apache:mainfrom 
h-vetinari:conda_sync
Sep 28, 2023
Merged
GH-37621: [Packaging][Conda] Sync conda recipes with feedstocks#37624
jorisvandenbossche merged 9 commits into
apache:mainfrom 
h-vetinari:conda_sync
Conversation
@h-vetinari
@h-vetinarih-vetinari
commented
Sep 8, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Syncing after the release of 13.0.0 + a couple of migrations (state as of conda-forge/arrow-cpp-feedstock#1168 & conda-forge/r-arrow-feedstock#68)
Relevant updates:
Also some further hardening of the activation scripts & clean-ups for dependencies & test skips.
@github-actions
github-actionsBot
commented
Sep 8, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #37621has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Sep 8, 2023
@h-vetinari
h-vetinari
commented
Sep 8, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
Sep 8, 2023

Copy link


Copy Markdown

Revision: c04c157
Submitted crossbow builds: ursacomputing/crossbow @ actions-630421cd79
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cpu-r43Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cpu-r43Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-arm64-cpu-r43Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r42Azure
conda-osx-x64-cpu-r43Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting review
Awaiting review
labels
Sep 8, 2023
@h-vetinari
h-vetinari
commented
Sep 8, 2023

Copy link


Copy Markdown

ContributorAuthor
Something is not working out with the CUDA cross-compilation here:
[82/139] Linking CXX shared module release/_cuda.cpython-310-powerpc64le-linux-gnu.so
FAILED: release/_cuda.cpython-310-powerpc64le-linux-gnu.so : && $BUILD_PREFIX/bin/powerpc64le-conda-linux-gnu-c++ -fPIC -Wno-noexcept-type -Wall -fno-semantic-interposition -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -mcpu=power8 -mtune=power8 -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -isystem $PREFIX/include -fdebug-prefix-map=/build/apache-arrow_1694161537567/work=/usr/local/src/conda/pyarrow-14.0.0.dev124 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -isystem /usr/local/cuda/targets/ppc64le-linux/include -fdiagnostics-color=always -fno-omit-frame-pointer -Wno-unused-variable -Wno-maybe-uninitialized -O3 -DNDEBUG -O2 -ftree-vectorize -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--allow-shlib-undefined -Wl,-rpath,$PREFIX/lib -Wl,-rpath-link,$PREFIX/lib -L$PREFIX/lib -shared -o release/_cuda.cpython-310-powerpc64le-linux-gnu.so CMakeFiles/_cuda.dir/_cuda.cpp.o -Wl,-rpath,"\$ORIGIN" release/libarrow_python.so $PREFIX/lib/libarrow_cuda.so.1400.0.0 $PREFIX/lib/libarrow_dataset.so.1400.0.0 $PREFIX/lib/libarrow_acero.so.1400.0.0 $PREFIX/lib/libparquet.so.1400.0.0 $PREFIX/lib/libarrow.so.1400.0.0 /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so && :
$BUILD_PREFIX/powerpc64le-conda-linux-gnu/bin/ld: /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status

It does work in conda-forge, and CUDA on aarch/ppc is a bit more of a niche setup, so I guess we could also drop it here. For completeness, I looked at the diff for the build scripts, but I couldn't determine something that would touch upon where the CUDA_HOME would point -- the error is pretty clearly that we're not pointing to the ppc version of libcuda.so:
$BUILD_PREFIX/powerpc64le-conda-linux-gnu/bin/ld: /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so: error adding symbols: file in wrong format

@h-vetinari
h-vetinari
commented
Sep 8, 2023

Copy link


Copy Markdown

ContributorAuthor
Is it intended that gandiva now has a run-time dependence on libllvm?
 import pyarrow.gandiva
ImportError: libLLVM-15.so: cannot open shared object file: No such file or directory

Adding the respective host-dependence (+ respective run-export) is not hard, I'm just double-checking that this is intentional.
CC @pitrou@kou@raulcd@jorisvandenbossche
PS. This is a rare case where windows looks better than unix, only one test failure:
=========================== short test summary info ===========================
FAILED tests/parquet/test_basic.py::test_fastparquet_cross_compatibility - As...
= 1 failed, 7154 passed, 355 skipped, 12 deselected, 22 xfailed, 2 xpassed, 8 warnings in 302.94s (0:05:02) =

@kou
kou
commented
Sep 8, 2023

Copy link


Copy Markdown

Member
Yes. It's caused by #37412.

We use libLLVM.so by default on non-Windows. If you don't like it you can add -DARROW_LLVM_USE_SHARED=OFF to use static LLVM libraries.

(I think that libLLVM.so is better to reduce duplication.)
@h-vetinarih-vetinari
mentioned this pull request
Sep 8, 2023
 Closed

@h-vetinari
h-vetinari
commented
Sep 12, 2023

Copy link


Copy Markdown

ContributorAuthor
It does work in conda-forge, and CUDA on aarch/ppc is a bit more of a niche setup, so I guess we could also drop it here.
I started debugging this PR on conda-forge infrastructure in conda-forge/arrow-cpp-feedstock#1170, and it turns out that the problem hits us there as well. I'm not sure what changed since 13.0, but it seems something is now overriding (or not respecting) our CUDA_HOME, which is necessary to make cross-compilation work with CUDA...
@kou
kou
commented
Sep 13, 2023

Copy link


Copy Markdown

Member
Could you try -DCUDAToolkit_ROOT=${CUDA_HOME} instead of -DCUDA_TOOLKIT_ROOT_DIR=${CUDA_HOME}?
CUDA_TOOLKIT_ROOT_DIR is for old FindCUDA: https://cmake.org/cmake/help/latest/module/FindCUDA.html

We use FindCUDAToolkit for new CMake: https://cmake.org/cmake/help/latest/module/FindCUDAToolkit.html
@h-vetinari
h-vetinari
commented
Sep 13, 2023

Copy link


Copy Markdown

ContributorAuthor
Thanks for the quick response, that sounds like a very promising candidate! Rebased & retriggered conda-forge/arrow-cpp-feedstock#1170. Will sync back to this PR if passing.
@h-vetinarih-vetinari
mentioned this pull request
Sep 13, 2023

Closed

@github-actionsgithub-actionsBot
added

Component: Python

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Sep 17, 2023
h-vetinari
h-vetinari
commented
Sep 17, 2023
View reviewed changes
Comment threadpython/pyarrow/tests/test_fs.py
Outdated

Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml

Comment on lines

+389
 to 
+390

 # currently broken
 {% set tests_to_skip = tests_to_skip + " or test_fastparquet_cross_compatibility" %}
@h-vetinarih-vetinariSep 17, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't raised an issue for this yet, but this is consistently failing. Not sure what's the difference between our CI and the one here.
@jorisvandenbosschejorisvandenbosscheSep 21, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know a CI build log that shows the error for this? (the last ones will have this already skipped)

(to have an idea what is going on here)
@h-vetinarih-vetinariSep 21, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_____________________ test_fastparquet_cross_compatibility _____________________
tempdir = PosixPath('/tmp/pytest-of-conda/pytest-0/test_fastparquet_cross_compati0')
@pytest.mark.pandas
@pytest.mark.fastparquet
@pytest.mark.filterwarnings("ignore:RangeIndex:FutureWarning")
@pytest.mark.filterwarnings("ignore:tostring:DeprecationWarning:fastparquet")
def test_fastparquet_cross_compatibility(tempdir):
fp = pytest.importorskip('fastparquet')
df = pd.DataFrame(
{
"a": list("abc"),
"b": list(range(1, 4)),
"c": np.arange(4.0, 7.0, dtype="float64"),
"d": [True, False, True],
"e": pd.date_range("20130101", periods=3),
"f": pd.Categorical(["a", "b", "a"]),
# fastparquet writes list as BYTE_ARRAY JSON, so no roundtrip
# "g": [[1, 2], None, [1, 2, 3]],
}
)
table = pa.table(df)
# Arrow -> fastparquet
file_arrow = str(tempdir / "cross_compat_arrow.parquet")
pq.write_table(table, file_arrow, compression=None)
fp_file = fp.ParquetFile(file_arrow)
df_fp = fp_file.to_pandas()
> tm.assert_frame_equal(df, df_fp)
pyarrow/tests/parquet/test_basic.py:741: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ testing.pyx:55: in pandas._libs.testing.assert_almost_equal
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ???
E AssertionError: DataFrame.iloc[:, 3] (column name="d") are different
E E DataFrame.iloc[:, 3] (column name="d") values are different (66.66667 %)
E [index]: [0, 1, 2]
E [left]: [True, False, True]
E [right]: [False, False, False]
E At positional index 0, first diff: True != False
testing.pyx:173: AssertionError

@jorisvandenbosschejorisvandenbosscheSep 25, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Searching for fastparquet in our code base, I assume that we actually don't have any CI build that includes it ... So we are just not running that test anywhere, whoops.
Now, the cross-compatibility is also tested in the pandas test suite, so maybe it's not too important to have it here as well. Anyway, opened an issue at #37853
@jorisvandenbosschejorisvandenbosscheSep 25, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although fastparquet reading a column of boolens wrongly for a pyarrow-written file seems a quite serious issue ..

Now, similar data is used in the pandas test suite, where this is passing.
@h-vetinarih-vetinariSep 25, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for raising the issue. I agree it does looks potentially serious, but I hadn't looked too closely because it was just one test. The first thing would be to add testing here in CI. I think this PR is fine as is (I try to keep the test skips to an absolute minimum and remove them whenever possible; that then gets picked up by the next recipe sync anyway)
@jorisvandenbosschejorisvandenbosscheSep 25, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took a closer look at it (could reproduce this locally), and so actually it was good to do so, as this is a problem in fastparquet reading parquet files generated by latest pyarrow, see details at #37853 (comment)
But yes, so skipping for now is perfectly fine.
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
Outdated

@h-vetinari
h-vetinari
commented
Sep 17, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
Sep 17, 2023

Copy link


Copy Markdown

Revision: d7af614
Submitted crossbow builds: ursacomputing/crossbow @ actions-20144be8d4
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cpu-r43Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cpu-r43Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-arm64-cpu-r43Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r42Azure
conda-osx-x64-cpu-r43Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@github-actions
github-actionsBot
commented
Sep 21, 2023

Copy link


Copy Markdown

Revision: 53a4b70
Submitted crossbow builds: ursacomputing/crossbow @ actions-80a2017be8
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cpu-r43Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cpu-r43Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-arm64-cpu-r43Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r42Azure
conda-osx-x64-cpu-r43Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@h-vetinari
h-vetinari
commented
Sep 25, 2023

Copy link


Copy Markdown

ContributorAuthor
Failures:
linux-64
Failed during artefact upload -- irrelevant
linux-aarch64
Connection lost to agent -- irrelevant
osx-64
=================================== FAILURES ===================================
_____________ test_debug_memory_pool_disabled[system_memory_pool] ______________
pool_factory = <built-in function system_memory_pool>
@pytest.mark.parametrize('pool_factory', supported_factories())
def test_debug_memory_pool_disabled(pool_factory):
res = run_debug_memory_pool(pool_factory.__name__, "")
# The subprocess either returned successfully or was killed by a signal
# (due to writing out of bounds), depending on the underlying allocator.
if os.name == "posix":
assert res.returncode <= 0
else:
res.check_returncode()
> assert res.stderr == ""
E AssertionError: assert 'libc++abi: P...ion called!\n' == ''
E + libc++abi: Pure virtual function called!
pyarrow/tests/test_memory.py:255: AssertionError
----------------------------- Captured stderr call -----------------------------
libc++abi: Pure virtual function called!

I can skip the respective test on osx if that's what people prefer. Otherwise this PR should be ready.
h-vetinari
h-vetinari
commented
Sep 25, 2023
View reviewed changes
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting change review
Awaiting change review
labels
Sep 25, 2023
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Sep 25, 2023
h-vetinari
h-vetinari
commented
Sep 25, 2023
View reviewed changes
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
Outdated

@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Sep 25, 2023

Closed

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting change review
Awaiting change review
labels
Sep 25, 2023
@h-vetinari
h-vetinari
commented
Sep 28, 2023

Copy link


Copy Markdown

ContributorAuthor
Anything else to do here?
@jorisvandenbossche
jorisvandenbossche
commented
Sep 28, 2023

Copy link


Copy Markdown

Member
I don't think so, thanks for the ping, and for all the work here!
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting changes
Awaiting changes
labels
Sep 28, 2023
@jorisvandenbossche
jorisvandenbossche
merged commit c9674bc
into
apache:mainSep 28, 2023
@jorisvandenbosschejorisvandenbossche
removed
the 
awaiting merge
Awaiting merge
label
Sep 28, 2023
@h-vetinari
h-vetinari
deleted the

conda_sync

branch
September 28, 2023 11:58
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Sep 28, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit c9674bc.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@h-vetinari@loicalleyne
apacheGH-37621: [Packaging][Conda] Sync conda recipes with feedstocks (…
6e5baa3
…apache#37624)
Syncing after the release of 13.0.0 + a couple of migrations (state as of conda-forge/arrow-cpp-feedstock#1168 & conda-forge/r-arrow-feedstock#68)
Relevant updates:
* we're not building twice for different protobuf versions anymore
* new abseil version (fixesapache#36908)
* we've finally upgraded the aws-sdk to 1.11
* the default R versions (on unix) are now 4.2 & 4.3.
Also some further hardening of the activation scripts & clean-ups for dependencies & test skips.
* Closes: apache#37621
Lead-authored-by: H. Vetinari <h.vetinari@gmx.com>
Co-authored-by: h-vetinari <h.vetinari@gmx.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
dgreiss pushed a commit
to dgreiss/arrow
that referenced
this pull request
Feb 19, 2024
@h-vetinari@dgreiss
apacheGH-37621: [Packaging][Conda] Sync conda recipes with feedstocks (…
e4438e8
…apache#37624)
Syncing after the release of 13.0.0 + a couple of migrations (state as of conda-forge/arrow-cpp-feedstock#1168 & conda-forge/r-arrow-feedstock#68)
Relevant updates:
* we're not building twice for different protobuf versions anymore
* new abseil version (fixesapache#36908)
* we've finally upgraded the aws-sdk to 1.11
* the default R versions (on unix) are now 4.2 & 4.3.
Also some further hardening of the activation scripts & clean-ups for dependencies & test skips.
* Closes: apache#37621
Lead-authored-by: H. Vetinari <h.vetinari@gmx.com>
Co-authored-by: h-vetinari <h.vetinari@gmx.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
May 17, 2024
 Merged

5 tasks

This was referenced Jun 11, 2024

Closed

 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@raulcdraulcdraulcd left review comments
@pitroupitroupitrou left review comments
@jorisvandenbosschejorisvandenbosschejorisvandenbossche approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@koukouAwaiting requested review from koukou is a code owner

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Sync conda-forge recipe

[Python][FlightRPC] Tests segfault on OSX in conda-forge


5 participants

@h-vetinari@kou@jorisvandenbossche@raulcd@pitrou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
GH-37621: [Packaging][Conda] Sync conda recipes with feedstocks - #37624
Merged
jorisvandenbossche merged 9 commits into
apache:mainfrom 
h-vetinari:conda_sync
Sep 28, 2023
Merged
GH-37621: [Packaging][Conda] Sync conda recipes with feedstocks#37624
jorisvandenbossche merged 9 commits into
apache:mainfrom 
h-vetinari:conda_sync
Conversation
@h-vetinari
@h-vetinarih-vetinari
commented
Sep 8, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Syncing after the release of 13.0.0 + a couple of migrations (state as of conda-forge/arrow-cpp-feedstock#1168 & conda-forge/r-arrow-feedstock#68)
Relevant updates:
Also some further hardening of the activation scripts & clean-ups for dependencies & test skips.
@github-actions
github-actionsBot
commented
Sep 8, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #37621has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
Sep 8, 2023
@h-vetinari
h-vetinari
commented
Sep 8, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
Sep 8, 2023

Copy link


Copy Markdown

Revision: c04c157
Submitted crossbow builds: ursacomputing/crossbow @ actions-630421cd79
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cpu-r43Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cpu-r43Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-arm64-cpu-r43Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r42Azure
conda-osx-x64-cpu-r43Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting review
Awaiting review
labels
Sep 8, 2023
@h-vetinari
h-vetinari
commented
Sep 8, 2023

Copy link


Copy Markdown

ContributorAuthor
Something is not working out with the CUDA cross-compilation here:
[82/139] Linking CXX shared module release/_cuda.cpython-310-powerpc64le-linux-gnu.so
FAILED: release/_cuda.cpython-310-powerpc64le-linux-gnu.so : && $BUILD_PREFIX/bin/powerpc64le-conda-linux-gnu-c++ -fPIC -Wno-noexcept-type -Wall -fno-semantic-interposition -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -mcpu=power8 -mtune=power8 -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -isystem $PREFIX/include -fdebug-prefix-map=/build/apache-arrow_1694161537567/work=/usr/local/src/conda/pyarrow-14.0.0.dev124 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -isystem /usr/local/cuda/targets/ppc64le-linux/include -fdiagnostics-color=always -fno-omit-frame-pointer -Wno-unused-variable -Wno-maybe-uninitialized -O3 -DNDEBUG -O2 -ftree-vectorize -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--allow-shlib-undefined -Wl,-rpath,$PREFIX/lib -Wl,-rpath-link,$PREFIX/lib -L$PREFIX/lib -shared -o release/_cuda.cpython-310-powerpc64le-linux-gnu.so CMakeFiles/_cuda.dir/_cuda.cpp.o -Wl,-rpath,"\$ORIGIN" release/libarrow_python.so $PREFIX/lib/libarrow_cuda.so.1400.0.0 $PREFIX/lib/libarrow_dataset.so.1400.0.0 $PREFIX/lib/libarrow_acero.so.1400.0.0 $PREFIX/lib/libparquet.so.1400.0.0 $PREFIX/lib/libarrow.so.1400.0.0 /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so && :
$BUILD_PREFIX/powerpc64le-conda-linux-gnu/bin/ld: /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status

It does work in conda-forge, and CUDA on aarch/ppc is a bit more of a niche setup, so I guess we could also drop it here. For completeness, I looked at the diff for the build scripts, but I couldn't determine something that would touch upon where the CUDA_HOME would point -- the error is pretty clearly that we're not pointing to the ppc version of libcuda.so:
$BUILD_PREFIX/powerpc64le-conda-linux-gnu/bin/ld: /usr/local/cuda-11.2/targets/x86_64-linux/lib/stubs/libcuda.so: error adding symbols: file in wrong format

@h-vetinari
h-vetinari
commented
Sep 8, 2023

Copy link


Copy Markdown

ContributorAuthor
Is it intended that gandiva now has a run-time dependence on libllvm?
 import pyarrow.gandiva
ImportError: libLLVM-15.so: cannot open shared object file: No such file or directory

Adding the respective host-dependence (+ respective run-export) is not hard, I'm just double-checking that this is intentional.
CC @pitrou@kou@raulcd@jorisvandenbossche
PS. This is a rare case where windows looks better than unix, only one test failure:
=========================== short test summary info ===========================
FAILED tests/parquet/test_basic.py::test_fastparquet_cross_compatibility - As...
= 1 failed, 7154 passed, 355 skipped, 12 deselected, 22 xfailed, 2 xpassed, 8 warnings in 302.94s (0:05:02) =

@kou
kou
commented
Sep 8, 2023

Copy link


Copy Markdown

Member
Yes. It's caused by #37412.

We use libLLVM.so by default on non-Windows. If you don't like it you can add -DARROW_LLVM_USE_SHARED=OFF to use static LLVM libraries.

(I think that libLLVM.so is better to reduce duplication.)
@h-vetinarih-vetinari
mentioned this pull request
Sep 8, 2023
 Closed

@h-vetinari
h-vetinari
commented
Sep 12, 2023

Copy link


Copy Markdown

ContributorAuthor
It does work in conda-forge, and CUDA on aarch/ppc is a bit more of a niche setup, so I guess we could also drop it here.
I started debugging this PR on conda-forge infrastructure in conda-forge/arrow-cpp-feedstock#1170, and it turns out that the problem hits us there as well. I'm not sure what changed since 13.0, but it seems something is now overriding (or not respecting) our CUDA_HOME, which is necessary to make cross-compilation work with CUDA...
@kou
kou
commented
Sep 13, 2023

Copy link


Copy Markdown

Member
Could you try -DCUDAToolkit_ROOT=${CUDA_HOME} instead of -DCUDA_TOOLKIT_ROOT_DIR=${CUDA_HOME}?
CUDA_TOOLKIT_ROOT_DIR is for old FindCUDA: https://cmake.org/cmake/help/latest/module/FindCUDA.html

We use FindCUDAToolkit for new CMake: https://cmake.org/cmake/help/latest/module/FindCUDAToolkit.html
@h-vetinari
h-vetinari
commented
Sep 13, 2023

Copy link


Copy Markdown

ContributorAuthor
Thanks for the quick response, that sounds like a very promising candidate! Rebased & retriggered conda-forge/arrow-cpp-feedstock#1170. Will sync back to this PR if passing.
@h-vetinarih-vetinari
mentioned this pull request
Sep 13, 2023

Closed

@github-actionsgithub-actionsBot
added

Component: Python

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Sep 17, 2023
h-vetinari
h-vetinari
commented
Sep 17, 2023
View reviewed changes
Comment threadpython/pyarrow/tests/test_fs.py
Outdated

Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml

Comment on lines

+389
 to 
+390

 # currently broken
 {% set tests_to_skip = tests_to_skip + " or test_fastparquet_cross_compatibility" %}
@h-vetinarih-vetinariSep 17, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't raised an issue for this yet, but this is consistently failing. Not sure what's the difference between our CI and the one here.
@jorisvandenbosschejorisvandenbosscheSep 21, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know a CI build log that shows the error for this? (the last ones will have this already skipped)

(to have an idea what is going on here)
@h-vetinarih-vetinariSep 21, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_____________________ test_fastparquet_cross_compatibility _____________________
tempdir = PosixPath('/tmp/pytest-of-conda/pytest-0/test_fastparquet_cross_compati0')
@pytest.mark.pandas
@pytest.mark.fastparquet
@pytest.mark.filterwarnings("ignore:RangeIndex:FutureWarning")
@pytest.mark.filterwarnings("ignore:tostring:DeprecationWarning:fastparquet")
def test_fastparquet_cross_compatibility(tempdir):
fp = pytest.importorskip('fastparquet')
df = pd.DataFrame(
{
"a": list("abc"),
"b": list(range(1, 4)),
"c": np.arange(4.0, 7.0, dtype="float64"),
"d": [True, False, True],
"e": pd.date_range("20130101", periods=3),
"f": pd.Categorical(["a", "b", "a"]),
# fastparquet writes list as BYTE_ARRAY JSON, so no roundtrip
# "g": [[1, 2], None, [1, 2, 3]],
}
)
table = pa.table(df)
# Arrow -> fastparquet
file_arrow = str(tempdir / "cross_compat_arrow.parquet")
pq.write_table(table, file_arrow, compression=None)
fp_file = fp.ParquetFile(file_arrow)
df_fp = fp_file.to_pandas()
> tm.assert_frame_equal(df, df_fp)
pyarrow/tests/parquet/test_basic.py:741: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ testing.pyx:55: in pandas._libs.testing.assert_almost_equal
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ???
E AssertionError: DataFrame.iloc[:, 3] (column name="d") are different
E E DataFrame.iloc[:, 3] (column name="d") values are different (66.66667 %)
E [index]: [0, 1, 2]
E [left]: [True, False, True]
E [right]: [False, False, False]
E At positional index 0, first diff: True != False
testing.pyx:173: AssertionError

@jorisvandenbosschejorisvandenbosscheSep 25, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Searching for fastparquet in our code base, I assume that we actually don't have any CI build that includes it ... So we are just not running that test anywhere, whoops.
Now, the cross-compatibility is also tested in the pandas test suite, so maybe it's not too important to have it here as well. Anyway, opened an issue at #37853
@jorisvandenbosschejorisvandenbosscheSep 25, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although fastparquet reading a column of boolens wrongly for a pyarrow-written file seems a quite serious issue ..

Now, similar data is used in the pandas test suite, where this is passing.
@h-vetinarih-vetinariSep 25, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for raising the issue. I agree it does looks potentially serious, but I hadn't looked too closely because it was just one test. The first thing would be to add testing here in CI. I think this PR is fine as is (I try to keep the test skips to an absolute minimum and remove them whenever possible; that then gets picked up by the next recipe sync anyway)
@jorisvandenbosschejorisvandenbosscheSep 25, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took a closer look at it (could reproduce this locally), and so actually it was good to do so, as this is a problem in fastparquet reading parquet files generated by latest pyarrow, see details at #37853 (comment)
But yes, so skipping for now is perfectly fine.
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
Outdated

@h-vetinari
h-vetinari
commented
Sep 17, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
Sep 17, 2023

Copy link


Copy Markdown

Revision: d7af614
Submitted crossbow builds: ursacomputing/crossbow @ actions-20144be8d4
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cpu-r43Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cpu-r43Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-arm64-cpu-r43Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r42Azure
conda-osx-x64-cpu-r43Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@github-actions
github-actionsBot
commented
Sep 21, 2023

Copy link


Copy Markdown

Revision: 53a4b70
Submitted crossbow builds: ursacomputing/crossbow @ actions-80a2017be8
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cpu-r43Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cpu-r43Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-arm64-cpu-r43Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r42Azure
conda-osx-x64-cpu-r43Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@h-vetinari
h-vetinari
commented
Sep 25, 2023

Copy link


Copy Markdown

ContributorAuthor
Failures:
linux-64
Failed during artefact upload -- irrelevant
linux-aarch64
Connection lost to agent -- irrelevant
osx-64
=================================== FAILURES ===================================
_____________ test_debug_memory_pool_disabled[system_memory_pool] ______________
pool_factory = <built-in function system_memory_pool>
@pytest.mark.parametrize('pool_factory', supported_factories())
def test_debug_memory_pool_disabled(pool_factory):
res = run_debug_memory_pool(pool_factory.__name__, "")
# The subprocess either returned successfully or was killed by a signal
# (due to writing out of bounds), depending on the underlying allocator.
if os.name == "posix":
assert res.returncode <= 0
else:
res.check_returncode()
> assert res.stderr == ""
E AssertionError: assert 'libc++abi: P...ion called!\n' == ''
E + libc++abi: Pure virtual function called!
pyarrow/tests/test_memory.py:255: AssertionError
----------------------------- Captured stderr call -----------------------------
libc++abi: Pure virtual function called!

I can skip the respective test on osx if that's what people prefer. Otherwise this PR should be ready.
h-vetinari
h-vetinari
commented
Sep 25, 2023
View reviewed changes
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting change review
Awaiting change review
labels
Sep 25, 2023
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
and removed

awaiting changes
Awaiting changes
labels
Sep 25, 2023
h-vetinari
h-vetinari
commented
Sep 25, 2023
View reviewed changes
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
Outdated

@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Sep 25, 2023

Closed

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting change review
Awaiting change review
labels
Sep 25, 2023
@h-vetinari
h-vetinari
commented
Sep 28, 2023

Copy link


Copy Markdown

ContributorAuthor
Anything else to do here?
@jorisvandenbossche
jorisvandenbossche
commented
Sep 28, 2023

Copy link


Copy Markdown

Member
I don't think so, thanks for the ping, and for all the work here!
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting changes
Awaiting changes
labels
Sep 28, 2023
@jorisvandenbossche
jorisvandenbossche
merged commit c9674bc
into
apache:mainSep 28, 2023
@jorisvandenbosschejorisvandenbossche
removed
the 
awaiting merge
Awaiting merge
label
Sep 28, 2023
@h-vetinari
h-vetinari
deleted the

conda_sync

branch
September 28, 2023 11:58
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Sep 28, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit c9674bc.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@h-vetinari@loicalleyne
apacheGH-37621: [Packaging][Conda] Sync conda recipes with feedstocks (…
6e5baa3
…apache#37624)
Syncing after the release of 13.0.0 + a couple of migrations (state as of conda-forge/arrow-cpp-feedstock#1168 & conda-forge/r-arrow-feedstock#68)
Relevant updates:
* we're not building twice for different protobuf versions anymore
* new abseil version (fixesapache#36908)
* we've finally upgraded the aws-sdk to 1.11
* the default R versions (on unix) are now 4.2 & 4.3.
Also some further hardening of the activation scripts & clean-ups for dependencies & test skips.
* Closes: apache#37621
Lead-authored-by: H. Vetinari <h.vetinari@gmx.com>
Co-authored-by: h-vetinari <h.vetinari@gmx.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
dgreiss pushed a commit
to dgreiss/arrow
that referenced
this pull request
Feb 19, 2024
@h-vetinari@dgreiss
apacheGH-37621: [Packaging][Conda] Sync conda recipes with feedstocks (…
e4438e8
…apache#37624)
Syncing after the release of 13.0.0 + a couple of migrations (state as of conda-forge/arrow-cpp-feedstock#1168 & conda-forge/r-arrow-feedstock#68)
Relevant updates:
* we're not building twice for different protobuf versions anymore
* new abseil version (fixesapache#36908)
* we've finally upgraded the aws-sdk to 1.11
* the default R versions (on unix) are now 4.2 & 4.3.
Also some further hardening of the activation scripts & clean-ups for dependencies & test skips.
* Closes: apache#37621
Lead-authored-by: H. Vetinari <h.vetinari@gmx.com>
Co-authored-by: h-vetinari <h.vetinari@gmx.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
May 17, 2024
 Merged

5 tasks

This was referenced Jun 11, 2024

Closed

 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@raulcdraulcdraulcd left review comments
@pitroupitroupitrou left review comments
@jorisvandenbosschejorisvandenbosschejorisvandenbossche approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@koukouAwaiting requested review from koukou is a code owner

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Sync conda-forge recipe

[Python][FlightRPC] Tests segfault on OSX in conda-forge


5 participants

@h-vetinari@kou@jorisvandenbossche@raulcd@pitrou