Skip to content

GH-35658: [Packaging] Sync conda recipes with feedstocks - #35637

Merged
kou merged 10 commits into
apache:mainfrom
h-vetinari:conda_update
Jul 26, 2023
Merged

GH-35658: [Packaging] Sync conda recipes with feedstocks#35637
kou merged 10 commits into
apache:mainfrom
h-vetinari:conda_update

Conversation

@h-vetinari

@h-vetinarih-vetinari commented May 17, 2023•
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Contributor

Corresponds to conda-forge/arrow-cpp-feedstock#1053 for pyarrow (one failing test on our infra that needs debugging), as well as the state of the feedstock for r-arrow after conda-forge/r-arrow-feedstock#65

@github-actions

github-actionsBot commented May 17, 2023

Copy link
Copy Markdown

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

@h-vetinarih-vetinari mentioned this pull request May 17, 2023
Merged
@github-actionsgithub-actionsBot added the awaiting review Awaiting review label May 17, 2023
@h-vetinari

h-vetinari commented May 17, 2023

Copy link
Copy Markdown
ContributorAuthor

@github-actions crossbow submit -g conda

@github-actions

github-actionsBot commented May 17, 2023

Copy link
Copy Markdown

Revision: 8ef347b

Submitted crossbow builds: ursacomputing/crossbow @ actions-ca516ae62e

TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r41Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r41Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r41Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r41Azure
conda-osx-x64-cpu-r42Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure

@h-vetinari

h-vetinari commented May 17, 2023

Copy link
Copy Markdown
ContributorAuthor

Interestingly, the failure of test_total_bytes_allocated on linux from conda-forge/arrow-cpp-feedstock#1053 is not present here. But a weird error on windows:

E pyarrow.lib.ArrowInvalid: Cannot locate timezone 'UTC': Timezone database not found at "C:\Users\VssAdministrator\Downloads\tzdata"

@kou

kou commented May 17, 2023

Copy link
Copy Markdown
Member

I don't know why we didn't need timezone database for Windows jobs before but we have a shell script to download timezone database: https://github.com/apache/arrow/blob/main/ci/scripts/download_tz_database.sh

We're using the script on Windows too.

@h-vetinarih-vetinari changed the title Sync conda recipes with feedstocksGH-35658: Sync conda recipes with feedstocksMay 18, 2023
@github-actions

github-actionsBot commented May 18, 2023

Copy link
Copy Markdown

@github-actions

github-actionsBot commented May 18, 2023

Copy link
Copy Markdown

⚠️ GitHub issue #35658has been automatically assigned in GitHub to PR creator.

@h-vetinari

h-vetinari commented May 18, 2023

Copy link
Copy Markdown
ContributorAuthor

I don't know why we didn't need timezone database for Windows jobs before but we have a shell script to download timezone database:

Sounds like we should just add a dependency on the tzdata conda-forge package?

@assignUser

assignUser commented May 18, 2023

Copy link
Copy Markdown
Member

Sounds like we should just add a dependency on the tzdata conda-forge package?

That sound's good. Thanks for taking care of the sync work, much appreciated!

@kou

kou commented May 18, 2023

Copy link
Copy Markdown
Member

It may not work because it seems that tzdata doesn't include WindowsZones.xml.

@h-vetinari

h-vetinari commented May 18, 2023

Copy link
Copy Markdown
ContributorAuthor

It may not work because it seems that tzdata doesn't include WindowsZones.xml.

Why does arrow depend on this in the first place? It's the first package I see that doesn't work with tzdata. Also, based on the error message, it seems to be looking in the wrong place? I.e. C:\Users\VssAdministrator\Downloads\tzdata instead of %LIBRARY_PREFIX% (aside from the fact that inndeed, our tzdata does not containWindowsZones.xml).

The tzdata stuff is an arrow 13 problem though. More urgently, in conda-forge/arrow-cpp-feedstock#1053, I currently have a failing test test_total_bytes_allocated on arrow 12 (though not arrow 11 or before), but some commit between arrow 12 and current main seems to have fixed this - any ideas?

@raulcd

raulcd commented May 18, 2023•
edited
Loading

Copy link
Copy Markdown
Member

As a note, our nightly builds for wheels on Windows are also failing at the moment with the same error:

 E pyarrow.lib.ArrowInvalid: Cannot locate timezone 'UTC': Timezone database not found at "C:\Users\ContainerAdministrator\Downloads\tzdata"

The first failure is from 10 days ago since this commit was introduced:
053b5ee

@kou

kou commented May 19, 2023

Copy link
Copy Markdown
Member

Ah, sorry. I misread this problem.

It seems that WindowsZones.xml isn't related to this problem.

But tzdata path was hard-coded on Windows: https://github.com/apache/arrow/blob/main/cpp/src/arrow/vendored/datetime/tz.cpp#L265

So our code will not find tzdata installed by conda.

Could you open a new issue for tzdata related problem? We can work on this as a separated issue.

@raulcd

raulcd commented May 19, 2023•
edited
Loading

Copy link
Copy Markdown
Member

The datetime vendored library seems to have had a change lately: https://github.com/apache/arrow/pull/35612/files
And is quite important for the R release to be packed for 12.0.1. I am wondering whether this issue will have to be fixed for the 12.0.1 release too.
It won't as this test was added on a different ticket that won't be added to 12.0.1 .

edit: strikethrough

@h-vetinarih-vetinari mentioned this pull request May 24, 2023
Closed
@h-vetinari

h-vetinari commented May 24, 2023•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

The tzdata stuff is an arrow 13 problem though. More urgently, in conda-forge/arrow-cpp-feedstock#1053, I currently have a failing test test_total_bytes_allocated on arrow 12 (though not arrow 11 or before), but some commit between arrow 12 and current main seems to have fixed this - any ideas?

As I mentioned, the tzdata thing is much less urgent for us in conda-forge than the issue of that failing tests on the feedstock. I now opened a separate issue so this gets (hopefully...) more traction: #35728

@jorisvandenbossche

jorisvandenbossche commented May 24, 2023

Copy link
Copy Markdown
Member

Why does arrow depend on this in the first place? It's the first package I see that doesn't work with tzdata.

Arrow unfortunately can't work with the tz database from tzdata, because the underlying date library we use doesn't support the binary format, see #31472 and linked issues.

In other places we skip tests that requite tz database access on Windows, so that should be done for the new scalar test as well, I assume.

@jorisvandenbossche

jorisvandenbossche commented May 24, 2023

Copy link
Copy Markdown
Member

I opened #35735 to skip the test requiring a tz database on windows.

@h-vetinari

h-vetinari commented May 24, 2023•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Arrow unfortunately can't work with the tz database from tzdata, because the underlying date library we use doesn't support the binary format, see #31472 and linked issues.

I don't mind shipping arrow's vendored WindowsZones.xml in a pyarrow-specific location (i.e. site-packages/pyarrow; we can patch pyarrow to point to the right location). I think it would be better to do that than just fail? timezone-operations are pretty common (unless I misunderstand something), so not supporting that for the conda-forge builds would suck IMO

@h-vetinari

h-vetinari commented May 24, 2023

Copy link
Copy Markdown
ContributorAuthor

One issue with vendoring is that this doesn't make it easy to update the version / ensure you always have a recent tz database, also when installing an somewhat older pyarrow version.

Absolutely, that's why it would be nice if this could be handled through a "proper" dependency like tzdata. I haven't dived deep into #31472, but I don't understand why the format that tzdata has couldn't be transformed or otherwise made to work for arrow?

@jorisvandenbossche

jorisvandenbossche commented May 24, 2023

Copy link
Copy Markdown
Member

That's probably technically possible to implement such transformation, but someone still needs to do the work for that. A potential easier path forward is to update date.h to work with the compiled database on Windows (HowardHinnant/date#564)

@mattijnmattijn mentioned this pull request Jun 2, 2023
Merged
@h-vetinari

h-vetinari commented Jul 24, 2023

Copy link
Copy Markdown
ContributorAuthor

@github-actions crossbow submit -g conda

h-vetinari
h-vetinari commented Jul 24, 2023
View reviewed changes
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
- clangdev {{ llvm_version }}
- llvmdev {{ llvm_version }}
- cython
- cython <3

@h-vetinarih-vetinariJul 24, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to wait for #36745 (and backport it 12.0.1), but that PR has not landed as quickly as I hoped.

@github-actions

github-actionsBot commented Jul 24, 2023

Copy link
Copy Markdown

Revision: e9e6605

Submitted crossbow builds: ursacomputing/crossbow @ actions-b1a7f0eaed

TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r41Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r41Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r41Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r41Azure
conda-osx-x64-cpu-r42Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure

@github-actionsgithub-actionsBot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Jul 24, 2023
@h-vetinari

h-vetinari commented Jul 24, 2023

Copy link
Copy Markdown
ContributorAuthor

@github-actions crossbow submit -g conda

@github-actions

github-actionsBot commented Jul 24, 2023

Copy link
Copy Markdown

Revision: 17fc588

Submitted crossbow builds: ursacomputing/crossbow @ actions-dd4f8afbf0

TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r41Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r41Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r41Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r41Azure
conda-osx-x64-cpu-r42Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure

@h-vetinarih-vetinari mentioned this pull request Jul 24, 2023
Merged
@h-vetinari

h-vetinari commented Jul 25, 2023•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@kou@pitrou@raulcd@assignUser@jorisvandenbossche
This is green now and matches the state of the feedstocks. There are some more minor improvements on the way (e.g. a few - though not all - of the test skips can be removed), but I can also put those in the next sync PR after arrow 13 is officially out.

@koukou changed the title GH-35658: Sync conda recipes with feedstocksGH-35658: [Packaging] Sync conda recipes with feedstocksJul 25, 2023
assignUser
assignUser approved these changes Jul 26, 2023
View reviewed changes

@assignUserassignUser left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work, thanks @h-vetinari ! I will merge in a day or so if there are not objections

kou
kou approved these changes Jul 26, 2023
View reviewed changes

@koukou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Comment threaddev/tasks/conda-recipes/arrow-cpp/build-arrow.sh
EXTRA_CMAKE_ARGS="${EXTRA_CMAKE_ARGS} -DLLVM_LINK_EXECUTABLE=${BUILD_PREFIX}/bin/llvm-link"
sed -ie "s;protoc-gen-grpc.*$;protoc-gen-grpc=${BUILD_PREFIX}/bin/grpc_cpp_plugin\";g" ../src/arrow/flight/CMakeLists.txt
sed -ie 's;"--with-jemalloc-prefix\=je_arrow_";"--with-jemalloc-prefix\=je_arrow_" "--with-lg-page\=14";g' ../cmake_modules/ThirdpartyToolchain.cmake
sed -ie 's;"--with-jemalloc-prefix\=je_arrow_";"--with-jemalloc-prefix\=je_arrow_" "--with-lg-page\=16";g' ../cmake_modules/ThirdpartyToolchain.cmake

@koukouJul 26, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use -DARROW_JEMALLOC_LG_PAGE=16 instead of rewriting ThirdpartyToolchain.cmake.

@h-vetinarih-vetinariJul 26, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds great, I'll change that on the feedstock and in the next sync

@github-actionsgithub-actionsBot added awaiting merge Awaiting merge and removed awaiting committer review Awaiting committer review labels Jul 26, 2023
@kou
kou merged commit 8503c86 into apache:mainJul 26, 2023
@koukou removed the awaiting merge Awaiting merge label Jul 26, 2023
@h-vetinari
h-vetinari deleted the conda_update branch July 26, 2023 23:43
@conbench-apache-arrow

conbench-apache-arrowBot commented Aug 2, 2023

Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 8503c86.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

loicalleyne pushed a commit to loicalleyne/arrow that referenced this pull request Nov 13, 2023
@h-vetinari@loicalleyne
apacheGH-35658: [Packaging] Sync conda recipes with feedstocks (apach…
c1ea980
…e#35637)
Corresponds to conda-forge/arrow-cpp-feedstock#1053 for pyarrow (one failing test on our infra that needs debugging), as well as the state of the feedstock for r-arrow after conda-forge/r-arrow-feedstock#65
* Closes: apache#35658
Authored-by: H. Vetinari <h.vetinari@gmx.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
@h-vetinarih-vetinari mentioned this pull request Jun 12, 2024
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@koukoukou approved these changes
@raulcdraulcdraulcd approved these changes
@assignUserassignUserassignUser approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Packaging] Sync with conda-forge feedstocks

5 participants

@h-vetinari@kou@assignUser@raulcd@jorisvandenbossche
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
GH-35658: [Packaging] Sync conda recipes with feedstocks by h-vetinari · Pull Request #35637 · apache/arrow · GitHub
Skip to content
GH-35658: [Packaging] Sync conda recipes with feedstocks - #35637
Merged
kou merged 10 commits into
apache:mainfrom 
h-vetinari:conda_update
Jul 26, 2023
Merged
GH-35658: [Packaging] Sync conda recipes with feedstocks#35637
kou merged 10 commits into
apache:mainfrom 
h-vetinari:conda_update
Conversation
@h-vetinari
@h-vetinarih-vetinari
commented
May 17, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Contributor
Corresponds to conda-forge/arrow-cpp-feedstock#1053 for pyarrow (one failing test on our infra that needs debugging), as well as the state of the feedstock for r-arrow after conda-forge/r-arrow-feedstock#65
@github-actions
github-actionsBot
commented
May 17, 2023

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@h-vetinarih-vetinari
mentioned this pull request
May 17, 2023
 Merged

@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
May 17, 2023
@h-vetinari
h-vetinari
commented
May 17, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
May 17, 2023

Copy link


Copy Markdown

Revision: 8ef347b
Submitted crossbow builds: ursacomputing/crossbow @ actions-ca516ae62e
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r41Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r41Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r41Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r41Azure
conda-osx-x64-cpu-r42Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@h-vetinari
h-vetinari
commented
May 17, 2023

Copy link


Copy Markdown

ContributorAuthor
Interestingly, the failure of test_total_bytes_allocated on linux from conda-forge/arrow-cpp-feedstock#1053 is not present here. But a weird error on windows:
E pyarrow.lib.ArrowInvalid: Cannot locate timezone 'UTC': Timezone database not found at "C:\Users\VssAdministrator\Downloads\tzdata"

@kou
kou
commented
May 17, 2023

Copy link


Copy Markdown

Member
I don't know why we didn't need timezone database for Windows jobs before but we have a shell script to download timezone database: https://github.com/apache/arrow/blob/main/ci/scripts/download_tz_database.sh
We're using the script on Windows too.
@h-vetinarih-vetinari
changed the title
Sync conda recipes with feedstocksGH-35658: Sync conda recipes with feedstocksMay 18, 2023
@github-actions
github-actionsBot
commented
May 18, 2023

Copy link


Copy Markdown

@github-actions
github-actionsBot
commented
May 18, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #35658has been automatically assigned in GitHub to PR creator.
@h-vetinari
h-vetinari
commented
May 18, 2023

Copy link


Copy Markdown

ContributorAuthor
I don't know why we didn't need timezone database for Windows jobs before but we have a shell script to download timezone database:
Sounds like we should just add a dependency on the tzdata conda-forge package?
@assignUser
assignUser
commented
May 18, 2023

Copy link


Copy Markdown

Member
Sounds like we should just add a dependency on the tzdata conda-forge package?
That sound's good. Thanks for taking care of the sync work, much appreciated!
@kou
kou
commented
May 18, 2023

Copy link


Copy Markdown

Member
It may not work because it seems that tzdata doesn't include WindowsZones.xml.
@h-vetinari
h-vetinari
commented
May 18, 2023

Copy link


Copy Markdown

ContributorAuthor
It may not work because it seems that tzdata doesn't include WindowsZones.xml.
Why does arrow depend on this in the first place? It's the first package I see that doesn't work with tzdata. Also, based on the error message, it seems to be looking in the wrong place? I.e. C:\Users\VssAdministrator\Downloads\tzdata instead of %LIBRARY_PREFIX% (aside from the fact that inndeed, our tzdata does not containWindowsZones.xml).
The tzdata stuff is an arrow 13 problem though. More urgently, in conda-forge/arrow-cpp-feedstock#1053, I currently have a failing test test_total_bytes_allocated on arrow 12 (though not arrow 11 or before), but some commit between arrow 12 and current main seems to have fixed this - any ideas?
@raulcd
raulcd
commented
May 18, 2023•

edited

Loading

Copy link


Copy Markdown

Member
As a note, our nightly builds for wheels on Windows are also failing at the moment with the same error:
 E pyarrow.lib.ArrowInvalid: Cannot locate timezone 'UTC': Timezone database not found at "C:\Users\ContainerAdministrator\Downloads\tzdata"

The first failure is from 10 days ago since this commit was introduced:
053b5ee
@kou
kou
commented
May 19, 2023

Copy link


Copy Markdown

Member
Ah, sorry. I misread this problem.
It seems that WindowsZones.xml isn't related to this problem.
But tzdata path was hard-coded on Windows: https://github.com/apache/arrow/blob/main/cpp/src/arrow/vendored/datetime/tz.cpp#L265
So our code will not find tzdata installed by conda.
Could you open a new issue for tzdata related problem? We can work on this as a separated issue.
@raulcd
raulcd
commented
May 19, 2023•

edited

Loading

Copy link


Copy Markdown

Member
The datetime vendored library seems to have had a change lately: https://github.com/apache/arrow/pull/35612/files

And is quite important for the R release to be packed for 12.0.1. I am wondering whether this issue will have to be fixed for the 12.0.1 release too.

It won't as this test was added on a different ticket that won't be added to 12.0.1 .
edit: strikethrough
@h-vetinarih-vetinari
mentioned this pull request
May 24, 2023

Closed

@h-vetinari
h-vetinari
commented
May 24, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
The tzdata stuff is an arrow 13 problem though. More urgently, in conda-forge/arrow-cpp-feedstock#1053, I currently have a failing test test_total_bytes_allocated on arrow 12 (though not arrow 11 or before), but some commit between arrow 12 and current main seems to have fixed this - any ideas?
As I mentioned, the tzdata thing is much less urgent for us in conda-forge than the issue of that failing tests on the feedstock. I now opened a separate issue so this gets (hopefully...) more traction: #35728
@jorisvandenbossche
jorisvandenbossche
commented
May 24, 2023

Copy link


Copy Markdown

Member
Why does arrow depend on this in the first place? It's the first package I see that doesn't work with tzdata.
Arrow unfortunately can't work with the tz database from tzdata, because the underlying date library we use doesn't support the binary format, see #31472 and linked issues.
In other places we skip tests that requite tz database access on Windows, so that should be done for the new scalar test as well, I assume.
@jorisvandenbossche
jorisvandenbossche
commented
May 24, 2023

Copy link


Copy Markdown

Member
I opened #35735 to skip the test requiring a tz database on windows.
@h-vetinari
h-vetinari
commented
May 24, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Arrow unfortunately can't work with the tz database from tzdata, because the underlying date library we use doesn't support the binary format, see #31472 and linked issues.
I don't mind shipping arrow's vendored WindowsZones.xml in a pyarrow-specific location (i.e. site-packages/pyarrow; we can patch pyarrow to point to the right location). I think it would be better to do that than just fail? timezone-operations are pretty common (unless I misunderstand something), so not supporting that for the conda-forge builds would suck IMO
@h-vetinari
h-vetinari
commented
May 24, 2023

Copy link


Copy Markdown

ContributorAuthor
One issue with vendoring is that this doesn't make it easy to update the version / ensure you always have a recent tz database, also when installing an somewhat older pyarrow version.
Absolutely, that's why it would be nice if this could be handled through a "proper" dependency like tzdata. I haven't dived deep into #31472, but I don't understand why the format that tzdata has couldn't be transformed or otherwise made to work for arrow?
@jorisvandenbossche
jorisvandenbossche
commented
May 24, 2023

Copy link


Copy Markdown

Member
That's probably technically possible to implement such transformation, but someone still needs to do the work for that. A potential easier path forward is to update date.h to work with the compiled database on Windows (HowardHinnant/date#564)
@mattijnmattijn
mentioned this pull request
Jun 2, 2023
 Merged

@h-vetinari
h-vetinari
commented
Jul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
h-vetinari
h-vetinari
commented
Jul 24, 2023
View reviewed changes
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
 - clangdev {{ llvm_version }}
 - llvmdev {{ llvm_version }}
 - cython
 - cython <3
@h-vetinarih-vetinariJul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to wait for #36745 (and backport it 12.0.1), but that PR has not landed as quickly as I hoped.
@github-actions
github-actionsBot
commented
Jul 24, 2023

Copy link


Copy Markdown

Revision: e9e6605
Submitted crossbow builds: ursacomputing/crossbow @ actions-b1a7f0eaed
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r41Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r41Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r41Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r41Azure
conda-osx-x64-cpu-r42Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@github-actionsgithub-actionsBot
added

awaiting committer review
Awaiting committer review
and removed

awaiting review
Awaiting review
labels
Jul 24, 2023
@h-vetinari
h-vetinari
commented
Jul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
Jul 24, 2023

Copy link


Copy Markdown

Revision: 17fc588
Submitted crossbow builds: ursacomputing/crossbow @ actions-dd4f8afbf0
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r41Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r41Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r41Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r41Azure
conda-osx-x64-cpu-r42Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@h-vetinarih-vetinari
mentioned this pull request
Jul 24, 2023
 Merged

@h-vetinari
h-vetinari
commented
Jul 25, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou@pitrou@raulcd@assignUser@jorisvandenbossche

This is green now and matches the state of the feedstocks. There are some more minor improvements on the way (e.g. a few - though not all - of the test skips can be removed), but I can also put those in the next sync PR after arrow 13 is officially out.
@koukou
changed the title
GH-35658: Sync conda recipes with feedstocksGH-35658: [Packaging] Sync conda recipes with feedstocksJul 25, 2023
assignUser
assignUser
approved these changes
Jul 26, 2023
View reviewed changes
@assignUserassignUser
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work, thanks @h-vetinari ! I will merge in a day or so if there are not objections
kou
kou
approved these changes
Jul 26, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Comment threaddev/tasks/conda-recipes/arrow-cpp/build-arrow.sh
 EXTRA_CMAKE_ARGS="${EXTRA_CMAKE_ARGS} -DLLVM_LINK_EXECUTABLE=${BUILD_PREFIX}/bin/llvm-link"
 sed -ie "s;protoc-gen-grpc.*$;protoc-gen-grpc=${BUILD_PREFIX}/bin/grpc_cpp_plugin\";g" ../src/arrow/flight/CMakeLists.txt
 sed -ie 's;"--with-jemalloc-prefix\=je_arrow_";"--with-jemalloc-prefix\=je_arrow_" "--with-lg-page\=14";g' ../cmake_modules/ThirdpartyToolchain.cmake
 sed -ie 's;"--with-jemalloc-prefix\=je_arrow_";"--with-jemalloc-prefix\=je_arrow_" "--with-lg-page\=16";g' ../cmake_modules/ThirdpartyToolchain.cmake
@koukouJul 26, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use -DARROW_JEMALLOC_LG_PAGE=16 instead of rewriting ThirdpartyToolchain.cmake.
@h-vetinarih-vetinariJul 26, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds great, I'll change that on the feedstock and in the next sync
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting committer review
Awaiting committer review
labels
Jul 26, 2023
@kou
kou
merged commit 8503c86
into
apache:mainJul 26, 2023
@koukou
removed
the 
awaiting merge
Awaiting merge
label
Jul 26, 2023
@h-vetinari
h-vetinari
deleted the

conda_update

branch
July 26, 2023 23:43
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Aug 2, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 8503c86.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@h-vetinari@loicalleyne
apacheGH-35658: [Packaging] Sync conda recipes with feedstocks (apach…
c1ea980
…e#35637)
Corresponds to conda-forge/arrow-cpp-feedstock#1053 for pyarrow (one failing test on our infra that needs debugging), as well as the state of the feedstock for r-arrow after conda-forge/r-arrow-feedstock#65
* Closes: apache#35658
Authored-by: H. Vetinari <h.vetinari@gmx.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
@h-vetinarih-vetinari
mentioned this pull request
Jun 12, 2024
 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@raulcdraulcdraulcd approved these changes
@assignUserassignUserassignUser approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[Packaging] Sync with conda-forge feedstocks


5 participants

@h-vetinari@kou@assignUser@raulcd@jorisvandenbossche

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GH-35658: [Packaging] Sync conda recipes with feedstocks by h-vetinari · Pull Request #35637 · apache/arrow · GitHub
Skip to content
GH-35658: [Packaging] Sync conda recipes with feedstocks - #35637
Merged
kou merged 10 commits into
apache:mainfrom 
h-vetinari:conda_update
Jul 26, 2023
Merged
GH-35658: [Packaging] Sync conda recipes with feedstocks#35637
kou merged 10 commits into
apache:mainfrom 
h-vetinari:conda_update
Conversation
@h-vetinari
@h-vetinarih-vetinari
commented
May 17, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Contributor
Corresponds to conda-forge/arrow-cpp-feedstock#1053 for pyarrow (one failing test on our infra that needs debugging), as well as the state of the feedstock for r-arrow after conda-forge/r-arrow-feedstock#65
@github-actions
github-actionsBot
commented
May 17, 2023

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@h-vetinarih-vetinari
mentioned this pull request
May 17, 2023
 Merged

@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
May 17, 2023
@h-vetinari
h-vetinari
commented
May 17, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
May 17, 2023

Copy link


Copy Markdown

Revision: 8ef347b
Submitted crossbow builds: ursacomputing/crossbow @ actions-ca516ae62e
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r41Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r41Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r41Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r41Azure
conda-osx-x64-cpu-r42Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@h-vetinari
h-vetinari
commented
May 17, 2023

Copy link


Copy Markdown

ContributorAuthor
Interestingly, the failure of test_total_bytes_allocated on linux from conda-forge/arrow-cpp-feedstock#1053 is not present here. But a weird error on windows:
E pyarrow.lib.ArrowInvalid: Cannot locate timezone 'UTC': Timezone database not found at "C:\Users\VssAdministrator\Downloads\tzdata"

@kou
kou
commented
May 17, 2023

Copy link


Copy Markdown

Member
I don't know why we didn't need timezone database for Windows jobs before but we have a shell script to download timezone database: https://github.com/apache/arrow/blob/main/ci/scripts/download_tz_database.sh
We're using the script on Windows too.
@h-vetinarih-vetinari
changed the title
Sync conda recipes with feedstocksGH-35658: Sync conda recipes with feedstocksMay 18, 2023
@github-actions
github-actionsBot
commented
May 18, 2023

Copy link


Copy Markdown

@github-actions
github-actionsBot
commented
May 18, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #35658has been automatically assigned in GitHub to PR creator.
@h-vetinari
h-vetinari
commented
May 18, 2023

Copy link


Copy Markdown

ContributorAuthor
I don't know why we didn't need timezone database for Windows jobs before but we have a shell script to download timezone database:
Sounds like we should just add a dependency on the tzdata conda-forge package?
@assignUser
assignUser
commented
May 18, 2023

Copy link


Copy Markdown

Member
Sounds like we should just add a dependency on the tzdata conda-forge package?
That sound's good. Thanks for taking care of the sync work, much appreciated!
@kou
kou
commented
May 18, 2023

Copy link


Copy Markdown

Member
It may not work because it seems that tzdata doesn't include WindowsZones.xml.
@h-vetinari
h-vetinari
commented
May 18, 2023

Copy link


Copy Markdown

ContributorAuthor
It may not work because it seems that tzdata doesn't include WindowsZones.xml.
Why does arrow depend on this in the first place? It's the first package I see that doesn't work with tzdata. Also, based on the error message, it seems to be looking in the wrong place? I.e. C:\Users\VssAdministrator\Downloads\tzdata instead of %LIBRARY_PREFIX% (aside from the fact that inndeed, our tzdata does not containWindowsZones.xml).
The tzdata stuff is an arrow 13 problem though. More urgently, in conda-forge/arrow-cpp-feedstock#1053, I currently have a failing test test_total_bytes_allocated on arrow 12 (though not arrow 11 or before), but some commit between arrow 12 and current main seems to have fixed this - any ideas?
@raulcd
raulcd
commented
May 18, 2023•

edited

Loading

Copy link


Copy Markdown

Member
As a note, our nightly builds for wheels on Windows are also failing at the moment with the same error:
 E pyarrow.lib.ArrowInvalid: Cannot locate timezone 'UTC': Timezone database not found at "C:\Users\ContainerAdministrator\Downloads\tzdata"

The first failure is from 10 days ago since this commit was introduced:
053b5ee
@kou
kou
commented
May 19, 2023

Copy link


Copy Markdown

Member
Ah, sorry. I misread this problem.
It seems that WindowsZones.xml isn't related to this problem.
But tzdata path was hard-coded on Windows: https://github.com/apache/arrow/blob/main/cpp/src/arrow/vendored/datetime/tz.cpp#L265
So our code will not find tzdata installed by conda.
Could you open a new issue for tzdata related problem? We can work on this as a separated issue.
@raulcd
raulcd
commented
May 19, 2023•

edited

Loading

Copy link


Copy Markdown

Member
The datetime vendored library seems to have had a change lately: https://github.com/apache/arrow/pull/35612/files

And is quite important for the R release to be packed for 12.0.1. I am wondering whether this issue will have to be fixed for the 12.0.1 release too.

It won't as this test was added on a different ticket that won't be added to 12.0.1 .
edit: strikethrough
@h-vetinarih-vetinari
mentioned this pull request
May 24, 2023

Closed

@h-vetinari
h-vetinari
commented
May 24, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
The tzdata stuff is an arrow 13 problem though. More urgently, in conda-forge/arrow-cpp-feedstock#1053, I currently have a failing test test_total_bytes_allocated on arrow 12 (though not arrow 11 or before), but some commit between arrow 12 and current main seems to have fixed this - any ideas?
As I mentioned, the tzdata thing is much less urgent for us in conda-forge than the issue of that failing tests on the feedstock. I now opened a separate issue so this gets (hopefully...) more traction: #35728
@jorisvandenbossche
jorisvandenbossche
commented
May 24, 2023

Copy link


Copy Markdown

Member
Why does arrow depend on this in the first place? It's the first package I see that doesn't work with tzdata.
Arrow unfortunately can't work with the tz database from tzdata, because the underlying date library we use doesn't support the binary format, see #31472 and linked issues.
In other places we skip tests that requite tz database access on Windows, so that should be done for the new scalar test as well, I assume.
@jorisvandenbossche
jorisvandenbossche
commented
May 24, 2023

Copy link


Copy Markdown

Member
I opened #35735 to skip the test requiring a tz database on windows.
@h-vetinari
h-vetinari
commented
May 24, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Arrow unfortunately can't work with the tz database from tzdata, because the underlying date library we use doesn't support the binary format, see #31472 and linked issues.
I don't mind shipping arrow's vendored WindowsZones.xml in a pyarrow-specific location (i.e. site-packages/pyarrow; we can patch pyarrow to point to the right location). I think it would be better to do that than just fail? timezone-operations are pretty common (unless I misunderstand something), so not supporting that for the conda-forge builds would suck IMO
@h-vetinari
h-vetinari
commented
May 24, 2023

Copy link


Copy Markdown

ContributorAuthor
One issue with vendoring is that this doesn't make it easy to update the version / ensure you always have a recent tz database, also when installing an somewhat older pyarrow version.
Absolutely, that's why it would be nice if this could be handled through a "proper" dependency like tzdata. I haven't dived deep into #31472, but I don't understand why the format that tzdata has couldn't be transformed or otherwise made to work for arrow?
@jorisvandenbossche
jorisvandenbossche
commented
May 24, 2023

Copy link


Copy Markdown

Member
That's probably technically possible to implement such transformation, but someone still needs to do the work for that. A potential easier path forward is to update date.h to work with the compiled database on Windows (HowardHinnant/date#564)
@mattijnmattijn
mentioned this pull request
Jun 2, 2023
 Merged

@h-vetinari
h-vetinari
commented
Jul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
h-vetinari
h-vetinari
commented
Jul 24, 2023
View reviewed changes
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
 - clangdev {{ llvm_version }}
 - llvmdev {{ llvm_version }}
 - cython
 - cython <3
@h-vetinarih-vetinariJul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to wait for #36745 (and backport it 12.0.1), but that PR has not landed as quickly as I hoped.
@github-actions
github-actionsBot
commented
Jul 24, 2023

Copy link


Copy Markdown

Revision: e9e6605
Submitted crossbow builds: ursacomputing/crossbow @ actions-b1a7f0eaed
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r41Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r41Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r41Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r41Azure
conda-osx-x64-cpu-r42Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@github-actionsgithub-actionsBot
added

awaiting committer review
Awaiting committer review
and removed

awaiting review
Awaiting review
labels
Jul 24, 2023
@h-vetinari
h-vetinari
commented
Jul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
Jul 24, 2023

Copy link


Copy Markdown

Revision: 17fc588
Submitted crossbow builds: ursacomputing/crossbow @ actions-dd4f8afbf0
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r41Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r41Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r41Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r41Azure
conda-osx-x64-cpu-r42Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@h-vetinarih-vetinari
mentioned this pull request
Jul 24, 2023
 Merged

@h-vetinari
h-vetinari
commented
Jul 25, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou@pitrou@raulcd@assignUser@jorisvandenbossche

This is green now and matches the state of the feedstocks. There are some more minor improvements on the way (e.g. a few - though not all - of the test skips can be removed), but I can also put those in the next sync PR after arrow 13 is officially out.
@koukou
changed the title
GH-35658: Sync conda recipes with feedstocksGH-35658: [Packaging] Sync conda recipes with feedstocksJul 25, 2023
assignUser
assignUser
approved these changes
Jul 26, 2023
View reviewed changes
@assignUserassignUser
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work, thanks @h-vetinari ! I will merge in a day or so if there are not objections
kou
kou
approved these changes
Jul 26, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Comment threaddev/tasks/conda-recipes/arrow-cpp/build-arrow.sh
 EXTRA_CMAKE_ARGS="${EXTRA_CMAKE_ARGS} -DLLVM_LINK_EXECUTABLE=${BUILD_PREFIX}/bin/llvm-link"
 sed -ie "s;protoc-gen-grpc.*$;protoc-gen-grpc=${BUILD_PREFIX}/bin/grpc_cpp_plugin\";g" ../src/arrow/flight/CMakeLists.txt
 sed -ie 's;"--with-jemalloc-prefix\=je_arrow_";"--with-jemalloc-prefix\=je_arrow_" "--with-lg-page\=14";g' ../cmake_modules/ThirdpartyToolchain.cmake
 sed -ie 's;"--with-jemalloc-prefix\=je_arrow_";"--with-jemalloc-prefix\=je_arrow_" "--with-lg-page\=16";g' ../cmake_modules/ThirdpartyToolchain.cmake
@koukouJul 26, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use -DARROW_JEMALLOC_LG_PAGE=16 instead of rewriting ThirdpartyToolchain.cmake.
@h-vetinarih-vetinariJul 26, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds great, I'll change that on the feedstock and in the next sync
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting committer review
Awaiting committer review
labels
Jul 26, 2023
@kou
kou
merged commit 8503c86
into
apache:mainJul 26, 2023
@koukou
removed
the 
awaiting merge
Awaiting merge
label
Jul 26, 2023
@h-vetinari
h-vetinari
deleted the

conda_update

branch
July 26, 2023 23:43
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Aug 2, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 8503c86.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@h-vetinari@loicalleyne
apacheGH-35658: [Packaging] Sync conda recipes with feedstocks (apach…
c1ea980
…e#35637)
Corresponds to conda-forge/arrow-cpp-feedstock#1053 for pyarrow (one failing test on our infra that needs debugging), as well as the state of the feedstock for r-arrow after conda-forge/r-arrow-feedstock#65
* Closes: apache#35658
Authored-by: H. Vetinari <h.vetinari@gmx.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
@h-vetinarih-vetinari
mentioned this pull request
Jun 12, 2024
 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@raulcdraulcdraulcd approved these changes
@assignUserassignUserassignUser approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[Packaging] Sync with conda-forge feedstocks


5 participants

@h-vetinari@kou@assignUser@raulcd@jorisvandenbossche

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GH-35658: [Packaging] Sync conda recipes with feedstocks by h-vetinari · Pull Request #35637 · apache/arrow · GitHub
Skip to content
GH-35658: [Packaging] Sync conda recipes with feedstocks - #35637
Merged
kou merged 10 commits into
apache:mainfrom 
h-vetinari:conda_update
Jul 26, 2023
Merged
GH-35658: [Packaging] Sync conda recipes with feedstocks#35637
kou merged 10 commits into
apache:mainfrom 
h-vetinari:conda_update
Conversation
@h-vetinari
@h-vetinarih-vetinari
commented
May 17, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Contributor
Corresponds to conda-forge/arrow-cpp-feedstock#1053 for pyarrow (one failing test on our infra that needs debugging), as well as the state of the feedstock for r-arrow after conda-forge/r-arrow-feedstock#65
@github-actions
github-actionsBot
commented
May 17, 2023

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@h-vetinarih-vetinari
mentioned this pull request
May 17, 2023
 Merged

@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
May 17, 2023
@h-vetinari
h-vetinari
commented
May 17, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
May 17, 2023

Copy link


Copy Markdown

Revision: 8ef347b
Submitted crossbow builds: ursacomputing/crossbow @ actions-ca516ae62e
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r41Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r41Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r41Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r41Azure
conda-osx-x64-cpu-r42Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@h-vetinari
h-vetinari
commented
May 17, 2023

Copy link


Copy Markdown

ContributorAuthor
Interestingly, the failure of test_total_bytes_allocated on linux from conda-forge/arrow-cpp-feedstock#1053 is not present here. But a weird error on windows:
E pyarrow.lib.ArrowInvalid: Cannot locate timezone 'UTC': Timezone database not found at "C:\Users\VssAdministrator\Downloads\tzdata"

@kou
kou
commented
May 17, 2023

Copy link


Copy Markdown

Member
I don't know why we didn't need timezone database for Windows jobs before but we have a shell script to download timezone database: https://github.com/apache/arrow/blob/main/ci/scripts/download_tz_database.sh
We're using the script on Windows too.
@h-vetinarih-vetinari
changed the title
Sync conda recipes with feedstocksGH-35658: Sync conda recipes with feedstocksMay 18, 2023
@github-actions
github-actionsBot
commented
May 18, 2023

Copy link


Copy Markdown

@github-actions
github-actionsBot
commented
May 18, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #35658has been automatically assigned in GitHub to PR creator.
@h-vetinari
h-vetinari
commented
May 18, 2023

Copy link


Copy Markdown

ContributorAuthor
I don't know why we didn't need timezone database for Windows jobs before but we have a shell script to download timezone database:
Sounds like we should just add a dependency on the tzdata conda-forge package?
@assignUser
assignUser
commented
May 18, 2023

Copy link


Copy Markdown

Member
Sounds like we should just add a dependency on the tzdata conda-forge package?
That sound's good. Thanks for taking care of the sync work, much appreciated!
@kou
kou
commented
May 18, 2023

Copy link


Copy Markdown

Member
It may not work because it seems that tzdata doesn't include WindowsZones.xml.
@h-vetinari
h-vetinari
commented
May 18, 2023

Copy link


Copy Markdown

ContributorAuthor
It may not work because it seems that tzdata doesn't include WindowsZones.xml.
Why does arrow depend on this in the first place? It's the first package I see that doesn't work with tzdata. Also, based on the error message, it seems to be looking in the wrong place? I.e. C:\Users\VssAdministrator\Downloads\tzdata instead of %LIBRARY_PREFIX% (aside from the fact that inndeed, our tzdata does not containWindowsZones.xml).
The tzdata stuff is an arrow 13 problem though. More urgently, in conda-forge/arrow-cpp-feedstock#1053, I currently have a failing test test_total_bytes_allocated on arrow 12 (though not arrow 11 or before), but some commit between arrow 12 and current main seems to have fixed this - any ideas?
@raulcd
raulcd
commented
May 18, 2023•

edited

Loading

Copy link


Copy Markdown

Member
As a note, our nightly builds for wheels on Windows are also failing at the moment with the same error:
 E pyarrow.lib.ArrowInvalid: Cannot locate timezone 'UTC': Timezone database not found at "C:\Users\ContainerAdministrator\Downloads\tzdata"

The first failure is from 10 days ago since this commit was introduced:
053b5ee
@kou
kou
commented
May 19, 2023

Copy link


Copy Markdown

Member
Ah, sorry. I misread this problem.
It seems that WindowsZones.xml isn't related to this problem.
But tzdata path was hard-coded on Windows: https://github.com/apache/arrow/blob/main/cpp/src/arrow/vendored/datetime/tz.cpp#L265
So our code will not find tzdata installed by conda.
Could you open a new issue for tzdata related problem? We can work on this as a separated issue.
@raulcd
raulcd
commented
May 19, 2023•

edited

Loading

Copy link


Copy Markdown

Member
The datetime vendored library seems to have had a change lately: https://github.com/apache/arrow/pull/35612/files

And is quite important for the R release to be packed for 12.0.1. I am wondering whether this issue will have to be fixed for the 12.0.1 release too.

It won't as this test was added on a different ticket that won't be added to 12.0.1 .
edit: strikethrough
@h-vetinarih-vetinari
mentioned this pull request
May 24, 2023

Closed

@h-vetinari
h-vetinari
commented
May 24, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
The tzdata stuff is an arrow 13 problem though. More urgently, in conda-forge/arrow-cpp-feedstock#1053, I currently have a failing test test_total_bytes_allocated on arrow 12 (though not arrow 11 or before), but some commit between arrow 12 and current main seems to have fixed this - any ideas?
As I mentioned, the tzdata thing is much less urgent for us in conda-forge than the issue of that failing tests on the feedstock. I now opened a separate issue so this gets (hopefully...) more traction: #35728
@jorisvandenbossche
jorisvandenbossche
commented
May 24, 2023

Copy link


Copy Markdown

Member
Why does arrow depend on this in the first place? It's the first package I see that doesn't work with tzdata.
Arrow unfortunately can't work with the tz database from tzdata, because the underlying date library we use doesn't support the binary format, see #31472 and linked issues.
In other places we skip tests that requite tz database access on Windows, so that should be done for the new scalar test as well, I assume.
@jorisvandenbossche
jorisvandenbossche
commented
May 24, 2023

Copy link


Copy Markdown

Member
I opened #35735 to skip the test requiring a tz database on windows.
@h-vetinari
h-vetinari
commented
May 24, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Arrow unfortunately can't work with the tz database from tzdata, because the underlying date library we use doesn't support the binary format, see #31472 and linked issues.
I don't mind shipping arrow's vendored WindowsZones.xml in a pyarrow-specific location (i.e. site-packages/pyarrow; we can patch pyarrow to point to the right location). I think it would be better to do that than just fail? timezone-operations are pretty common (unless I misunderstand something), so not supporting that for the conda-forge builds would suck IMO
@h-vetinari
h-vetinari
commented
May 24, 2023

Copy link


Copy Markdown

ContributorAuthor
One issue with vendoring is that this doesn't make it easy to update the version / ensure you always have a recent tz database, also when installing an somewhat older pyarrow version.
Absolutely, that's why it would be nice if this could be handled through a "proper" dependency like tzdata. I haven't dived deep into #31472, but I don't understand why the format that tzdata has couldn't be transformed or otherwise made to work for arrow?
@jorisvandenbossche
jorisvandenbossche
commented
May 24, 2023

Copy link


Copy Markdown

Member
That's probably technically possible to implement such transformation, but someone still needs to do the work for that. A potential easier path forward is to update date.h to work with the compiled database on Windows (HowardHinnant/date#564)
@mattijnmattijn
mentioned this pull request
Jun 2, 2023
 Merged

@h-vetinari
h-vetinari
commented
Jul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
h-vetinari
h-vetinari
commented
Jul 24, 2023
View reviewed changes
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
 - clangdev {{ llvm_version }}
 - llvmdev {{ llvm_version }}
 - cython
 - cython <3
@h-vetinarih-vetinariJul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to wait for #36745 (and backport it 12.0.1), but that PR has not landed as quickly as I hoped.
@github-actions
github-actionsBot
commented
Jul 24, 2023

Copy link


Copy Markdown

Revision: e9e6605
Submitted crossbow builds: ursacomputing/crossbow @ actions-b1a7f0eaed
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r41Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r41Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r41Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r41Azure
conda-osx-x64-cpu-r42Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@github-actionsgithub-actionsBot
added

awaiting committer review
Awaiting committer review
and removed

awaiting review
Awaiting review
labels
Jul 24, 2023
@h-vetinari
h-vetinari
commented
Jul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
Jul 24, 2023

Copy link


Copy Markdown

Revision: 17fc588
Submitted crossbow builds: ursacomputing/crossbow @ actions-dd4f8afbf0
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r41Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r41Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r41Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r41Azure
conda-osx-x64-cpu-r42Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@h-vetinarih-vetinari
mentioned this pull request
Jul 24, 2023
 Merged

@h-vetinari
h-vetinari
commented
Jul 25, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou@pitrou@raulcd@assignUser@jorisvandenbossche

This is green now and matches the state of the feedstocks. There are some more minor improvements on the way (e.g. a few - though not all - of the test skips can be removed), but I can also put those in the next sync PR after arrow 13 is officially out.
@koukou
changed the title
GH-35658: Sync conda recipes with feedstocksGH-35658: [Packaging] Sync conda recipes with feedstocksJul 25, 2023
assignUser
assignUser
approved these changes
Jul 26, 2023
View reviewed changes
@assignUserassignUser
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work, thanks @h-vetinari ! I will merge in a day or so if there are not objections
kou
kou
approved these changes
Jul 26, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Comment threaddev/tasks/conda-recipes/arrow-cpp/build-arrow.sh
 EXTRA_CMAKE_ARGS="${EXTRA_CMAKE_ARGS} -DLLVM_LINK_EXECUTABLE=${BUILD_PREFIX}/bin/llvm-link"
 sed -ie "s;protoc-gen-grpc.*$;protoc-gen-grpc=${BUILD_PREFIX}/bin/grpc_cpp_plugin\";g" ../src/arrow/flight/CMakeLists.txt
 sed -ie 's;"--with-jemalloc-prefix\=je_arrow_";"--with-jemalloc-prefix\=je_arrow_" "--with-lg-page\=14";g' ../cmake_modules/ThirdpartyToolchain.cmake
 sed -ie 's;"--with-jemalloc-prefix\=je_arrow_";"--with-jemalloc-prefix\=je_arrow_" "--with-lg-page\=16";g' ../cmake_modules/ThirdpartyToolchain.cmake
@koukouJul 26, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use -DARROW_JEMALLOC_LG_PAGE=16 instead of rewriting ThirdpartyToolchain.cmake.
@h-vetinarih-vetinariJul 26, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds great, I'll change that on the feedstock and in the next sync
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting committer review
Awaiting committer review
labels
Jul 26, 2023
@kou
kou
merged commit 8503c86
into
apache:mainJul 26, 2023
@koukou
removed
the 
awaiting merge
Awaiting merge
label
Jul 26, 2023
@h-vetinari
h-vetinari
deleted the

conda_update

branch
July 26, 2023 23:43
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Aug 2, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 8503c86.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@h-vetinari@loicalleyne
apacheGH-35658: [Packaging] Sync conda recipes with feedstocks (apach…
c1ea980
…e#35637)
Corresponds to conda-forge/arrow-cpp-feedstock#1053 for pyarrow (one failing test on our infra that needs debugging), as well as the state of the feedstock for r-arrow after conda-forge/r-arrow-feedstock#65
* Closes: apache#35658
Authored-by: H. Vetinari <h.vetinari@gmx.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
@h-vetinarih-vetinari
mentioned this pull request
Jun 12, 2024
 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@raulcdraulcdraulcd approved these changes
@assignUserassignUserassignUser approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[Packaging] Sync with conda-forge feedstocks


5 participants

@h-vetinari@kou@assignUser@raulcd@jorisvandenbossche

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
GH-35658: [Packaging] Sync conda recipes with feedstocks by h-vetinari · Pull Request #35637 · apache/arrow · GitHub
Skip to content
GH-35658: [Packaging] Sync conda recipes with feedstocks - #35637
Merged
kou merged 10 commits into
apache:mainfrom 
h-vetinari:conda_update
Jul 26, 2023
Merged
GH-35658: [Packaging] Sync conda recipes with feedstocks#35637
kou merged 10 commits into
apache:mainfrom 
h-vetinari:conda_update
Conversation
@h-vetinari
@h-vetinarih-vetinari
commented
May 17, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Contributor
Corresponds to conda-forge/arrow-cpp-feedstock#1053 for pyarrow (one failing test on our infra that needs debugging), as well as the state of the feedstock for r-arrow after conda-forge/r-arrow-feedstock#65
@github-actions
github-actionsBot
commented
May 17, 2023

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@h-vetinarih-vetinari
mentioned this pull request
May 17, 2023
 Merged

@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
May 17, 2023
@h-vetinari
h-vetinari
commented
May 17, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
May 17, 2023

Copy link


Copy Markdown

Revision: 8ef347b
Submitted crossbow builds: ursacomputing/crossbow @ actions-ca516ae62e
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r41Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r41Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r41Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r41Azure
conda-osx-x64-cpu-r42Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@h-vetinari
h-vetinari
commented
May 17, 2023

Copy link


Copy Markdown

ContributorAuthor
Interestingly, the failure of test_total_bytes_allocated on linux from conda-forge/arrow-cpp-feedstock#1053 is not present here. But a weird error on windows:
E pyarrow.lib.ArrowInvalid: Cannot locate timezone 'UTC': Timezone database not found at "C:\Users\VssAdministrator\Downloads\tzdata"

@kou
kou
commented
May 17, 2023

Copy link


Copy Markdown

Member
I don't know why we didn't need timezone database for Windows jobs before but we have a shell script to download timezone database: https://github.com/apache/arrow/blob/main/ci/scripts/download_tz_database.sh
We're using the script on Windows too.
@h-vetinarih-vetinari
changed the title
Sync conda recipes with feedstocksGH-35658: Sync conda recipes with feedstocksMay 18, 2023
@github-actions
github-actionsBot
commented
May 18, 2023

Copy link


Copy Markdown

@github-actions
github-actionsBot
commented
May 18, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #35658has been automatically assigned in GitHub to PR creator.
@h-vetinari
h-vetinari
commented
May 18, 2023

Copy link


Copy Markdown

ContributorAuthor
I don't know why we didn't need timezone database for Windows jobs before but we have a shell script to download timezone database:
Sounds like we should just add a dependency on the tzdata conda-forge package?
@assignUser
assignUser
commented
May 18, 2023

Copy link


Copy Markdown

Member
Sounds like we should just add a dependency on the tzdata conda-forge package?
That sound's good. Thanks for taking care of the sync work, much appreciated!
@kou
kou
commented
May 18, 2023

Copy link


Copy Markdown

Member
It may not work because it seems that tzdata doesn't include WindowsZones.xml.
@h-vetinari
h-vetinari
commented
May 18, 2023

Copy link


Copy Markdown

ContributorAuthor
It may not work because it seems that tzdata doesn't include WindowsZones.xml.
Why does arrow depend on this in the first place? It's the first package I see that doesn't work with tzdata. Also, based on the error message, it seems to be looking in the wrong place? I.e. C:\Users\VssAdministrator\Downloads\tzdata instead of %LIBRARY_PREFIX% (aside from the fact that inndeed, our tzdata does not containWindowsZones.xml).
The tzdata stuff is an arrow 13 problem though. More urgently, in conda-forge/arrow-cpp-feedstock#1053, I currently have a failing test test_total_bytes_allocated on arrow 12 (though not arrow 11 or before), but some commit between arrow 12 and current main seems to have fixed this - any ideas?
@raulcd
raulcd
commented
May 18, 2023•

edited

Loading

Copy link


Copy Markdown

Member
As a note, our nightly builds for wheels on Windows are also failing at the moment with the same error:
 E pyarrow.lib.ArrowInvalid: Cannot locate timezone 'UTC': Timezone database not found at "C:\Users\ContainerAdministrator\Downloads\tzdata"

The first failure is from 10 days ago since this commit was introduced:
053b5ee
@kou
kou
commented
May 19, 2023

Copy link


Copy Markdown

Member
Ah, sorry. I misread this problem.
It seems that WindowsZones.xml isn't related to this problem.
But tzdata path was hard-coded on Windows: https://github.com/apache/arrow/blob/main/cpp/src/arrow/vendored/datetime/tz.cpp#L265
So our code will not find tzdata installed by conda.
Could you open a new issue for tzdata related problem? We can work on this as a separated issue.
@raulcd
raulcd
commented
May 19, 2023•

edited

Loading

Copy link


Copy Markdown

Member
The datetime vendored library seems to have had a change lately: https://github.com/apache/arrow/pull/35612/files

And is quite important for the R release to be packed for 12.0.1. I am wondering whether this issue will have to be fixed for the 12.0.1 release too.

It won't as this test was added on a different ticket that won't be added to 12.0.1 .
edit: strikethrough
@h-vetinarih-vetinari
mentioned this pull request
May 24, 2023

Closed

@h-vetinari
h-vetinari
commented
May 24, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
The tzdata stuff is an arrow 13 problem though. More urgently, in conda-forge/arrow-cpp-feedstock#1053, I currently have a failing test test_total_bytes_allocated on arrow 12 (though not arrow 11 or before), but some commit between arrow 12 and current main seems to have fixed this - any ideas?
As I mentioned, the tzdata thing is much less urgent for us in conda-forge than the issue of that failing tests on the feedstock. I now opened a separate issue so this gets (hopefully...) more traction: #35728
@jorisvandenbossche
jorisvandenbossche
commented
May 24, 2023

Copy link


Copy Markdown

Member
Why does arrow depend on this in the first place? It's the first package I see that doesn't work with tzdata.
Arrow unfortunately can't work with the tz database from tzdata, because the underlying date library we use doesn't support the binary format, see #31472 and linked issues.
In other places we skip tests that requite tz database access on Windows, so that should be done for the new scalar test as well, I assume.
@jorisvandenbossche
jorisvandenbossche
commented
May 24, 2023

Copy link


Copy Markdown

Member
I opened #35735 to skip the test requiring a tz database on windows.
@h-vetinari
h-vetinari
commented
May 24, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Arrow unfortunately can't work with the tz database from tzdata, because the underlying date library we use doesn't support the binary format, see #31472 and linked issues.
I don't mind shipping arrow's vendored WindowsZones.xml in a pyarrow-specific location (i.e. site-packages/pyarrow; we can patch pyarrow to point to the right location). I think it would be better to do that than just fail? timezone-operations are pretty common (unless I misunderstand something), so not supporting that for the conda-forge builds would suck IMO
@h-vetinari
h-vetinari
commented
May 24, 2023

Copy link


Copy Markdown

ContributorAuthor
One issue with vendoring is that this doesn't make it easy to update the version / ensure you always have a recent tz database, also when installing an somewhat older pyarrow version.
Absolutely, that's why it would be nice if this could be handled through a "proper" dependency like tzdata. I haven't dived deep into #31472, but I don't understand why the format that tzdata has couldn't be transformed or otherwise made to work for arrow?
@jorisvandenbossche
jorisvandenbossche
commented
May 24, 2023

Copy link


Copy Markdown

Member
That's probably technically possible to implement such transformation, but someone still needs to do the work for that. A potential easier path forward is to update date.h to work with the compiled database on Windows (HowardHinnant/date#564)
@mattijnmattijn
mentioned this pull request
Jun 2, 2023
 Merged

@h-vetinari
h-vetinari
commented
Jul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
h-vetinari
h-vetinari
commented
Jul 24, 2023
View reviewed changes
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
 - clangdev {{ llvm_version }}
 - llvmdev {{ llvm_version }}
 - cython
 - cython <3
@h-vetinarih-vetinariJul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to wait for #36745 (and backport it 12.0.1), but that PR has not landed as quickly as I hoped.
@github-actions
github-actionsBot
commented
Jul 24, 2023

Copy link


Copy Markdown

Revision: e9e6605
Submitted crossbow builds: ursacomputing/crossbow @ actions-b1a7f0eaed
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r41Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r41Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r41Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r41Azure
conda-osx-x64-cpu-r42Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@github-actionsgithub-actionsBot
added

awaiting committer review
Awaiting committer review
and removed

awaiting review
Awaiting review
labels
Jul 24, 2023
@h-vetinari
h-vetinari
commented
Jul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
Jul 24, 2023

Copy link


Copy Markdown

Revision: 17fc588
Submitted crossbow builds: ursacomputing/crossbow @ actions-dd4f8afbf0
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r41Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r41Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r41Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r41Azure
conda-osx-x64-cpu-r42Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@h-vetinarih-vetinari
mentioned this pull request
Jul 24, 2023
 Merged

@h-vetinari
h-vetinari
commented
Jul 25, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou@pitrou@raulcd@assignUser@jorisvandenbossche

This is green now and matches the state of the feedstocks. There are some more minor improvements on the way (e.g. a few - though not all - of the test skips can be removed), but I can also put those in the next sync PR after arrow 13 is officially out.
@koukou
changed the title
GH-35658: Sync conda recipes with feedstocksGH-35658: [Packaging] Sync conda recipes with feedstocksJul 25, 2023
assignUser
assignUser
approved these changes
Jul 26, 2023
View reviewed changes
@assignUserassignUser
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work, thanks @h-vetinari ! I will merge in a day or so if there are not objections
kou
kou
approved these changes
Jul 26, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Comment threaddev/tasks/conda-recipes/arrow-cpp/build-arrow.sh
 EXTRA_CMAKE_ARGS="${EXTRA_CMAKE_ARGS} -DLLVM_LINK_EXECUTABLE=${BUILD_PREFIX}/bin/llvm-link"
 sed -ie "s;protoc-gen-grpc.*$;protoc-gen-grpc=${BUILD_PREFIX}/bin/grpc_cpp_plugin\";g" ../src/arrow/flight/CMakeLists.txt
 sed -ie 's;"--with-jemalloc-prefix\=je_arrow_";"--with-jemalloc-prefix\=je_arrow_" "--with-lg-page\=14";g' ../cmake_modules/ThirdpartyToolchain.cmake
 sed -ie 's;"--with-jemalloc-prefix\=je_arrow_";"--with-jemalloc-prefix\=je_arrow_" "--with-lg-page\=16";g' ../cmake_modules/ThirdpartyToolchain.cmake
@koukouJul 26, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use -DARROW_JEMALLOC_LG_PAGE=16 instead of rewriting ThirdpartyToolchain.cmake.
@h-vetinarih-vetinariJul 26, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds great, I'll change that on the feedstock and in the next sync
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting committer review
Awaiting committer review
labels
Jul 26, 2023
@kou
kou
merged commit 8503c86
into
apache:mainJul 26, 2023
@koukou
removed
the 
awaiting merge
Awaiting merge
label
Jul 26, 2023
@h-vetinari
h-vetinari
deleted the

conda_update

branch
July 26, 2023 23:43
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Aug 2, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 8503c86.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@h-vetinari@loicalleyne
apacheGH-35658: [Packaging] Sync conda recipes with feedstocks (apach…
c1ea980
…e#35637)
Corresponds to conda-forge/arrow-cpp-feedstock#1053 for pyarrow (one failing test on our infra that needs debugging), as well as the state of the feedstock for r-arrow after conda-forge/r-arrow-feedstock#65
* Closes: apache#35658
Authored-by: H. Vetinari <h.vetinari@gmx.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
@h-vetinarih-vetinari
mentioned this pull request
Jun 12, 2024
 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@raulcdraulcdraulcd approved these changes
@assignUserassignUserassignUser approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[Packaging] Sync with conda-forge feedstocks


5 participants

@h-vetinari@kou@assignUser@raulcd@jorisvandenbossche

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GH-35658: [Packaging] Sync conda recipes with feedstocks by h-vetinari · Pull Request #35637 · apache/arrow · GitHub
Skip to content
GH-35658: [Packaging] Sync conda recipes with feedstocks - #35637
Merged
kou merged 10 commits into
apache:mainfrom 
h-vetinari:conda_update
Jul 26, 2023
Merged
GH-35658: [Packaging] Sync conda recipes with feedstocks#35637
kou merged 10 commits into
apache:mainfrom 
h-vetinari:conda_update
Conversation
@h-vetinari
@h-vetinarih-vetinari
commented
May 17, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Contributor
Corresponds to conda-forge/arrow-cpp-feedstock#1053 for pyarrow (one failing test on our infra that needs debugging), as well as the state of the feedstock for r-arrow after conda-forge/r-arrow-feedstock#65
@github-actions
github-actionsBot
commented
May 17, 2023

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@h-vetinarih-vetinari
mentioned this pull request
May 17, 2023
 Merged

@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
May 17, 2023
@h-vetinari
h-vetinari
commented
May 17, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
May 17, 2023

Copy link


Copy Markdown

Revision: 8ef347b
Submitted crossbow builds: ursacomputing/crossbow @ actions-ca516ae62e
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r41Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r41Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r41Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r41Azure
conda-osx-x64-cpu-r42Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@h-vetinari
h-vetinari
commented
May 17, 2023

Copy link


Copy Markdown

ContributorAuthor
Interestingly, the failure of test_total_bytes_allocated on linux from conda-forge/arrow-cpp-feedstock#1053 is not present here. But a weird error on windows:
E pyarrow.lib.ArrowInvalid: Cannot locate timezone 'UTC': Timezone database not found at "C:\Users\VssAdministrator\Downloads\tzdata"

@kou
kou
commented
May 17, 2023

Copy link


Copy Markdown

Member
I don't know why we didn't need timezone database for Windows jobs before but we have a shell script to download timezone database: https://github.com/apache/arrow/blob/main/ci/scripts/download_tz_database.sh
We're using the script on Windows too.
@h-vetinarih-vetinari
changed the title
Sync conda recipes with feedstocksGH-35658: Sync conda recipes with feedstocksMay 18, 2023
@github-actions
github-actionsBot
commented
May 18, 2023

Copy link


Copy Markdown

@github-actions
github-actionsBot
commented
May 18, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #35658has been automatically assigned in GitHub to PR creator.
@h-vetinari
h-vetinari
commented
May 18, 2023

Copy link


Copy Markdown

ContributorAuthor
I don't know why we didn't need timezone database for Windows jobs before but we have a shell script to download timezone database:
Sounds like we should just add a dependency on the tzdata conda-forge package?
@assignUser
assignUser
commented
May 18, 2023

Copy link


Copy Markdown

Member
Sounds like we should just add a dependency on the tzdata conda-forge package?
That sound's good. Thanks for taking care of the sync work, much appreciated!
@kou
kou
commented
May 18, 2023

Copy link


Copy Markdown

Member
It may not work because it seems that tzdata doesn't include WindowsZones.xml.
@h-vetinari
h-vetinari
commented
May 18, 2023

Copy link


Copy Markdown

ContributorAuthor
It may not work because it seems that tzdata doesn't include WindowsZones.xml.
Why does arrow depend on this in the first place? It's the first package I see that doesn't work with tzdata. Also, based on the error message, it seems to be looking in the wrong place? I.e. C:\Users\VssAdministrator\Downloads\tzdata instead of %LIBRARY_PREFIX% (aside from the fact that inndeed, our tzdata does not containWindowsZones.xml).
The tzdata stuff is an arrow 13 problem though. More urgently, in conda-forge/arrow-cpp-feedstock#1053, I currently have a failing test test_total_bytes_allocated on arrow 12 (though not arrow 11 or before), but some commit between arrow 12 and current main seems to have fixed this - any ideas?
@raulcd
raulcd
commented
May 18, 2023•

edited

Loading

Copy link


Copy Markdown

Member
As a note, our nightly builds for wheels on Windows are also failing at the moment with the same error:
 E pyarrow.lib.ArrowInvalid: Cannot locate timezone 'UTC': Timezone database not found at "C:\Users\ContainerAdministrator\Downloads\tzdata"

The first failure is from 10 days ago since this commit was introduced:
053b5ee
@kou
kou
commented
May 19, 2023

Copy link


Copy Markdown

Member
Ah, sorry. I misread this problem.
It seems that WindowsZones.xml isn't related to this problem.
But tzdata path was hard-coded on Windows: https://github.com/apache/arrow/blob/main/cpp/src/arrow/vendored/datetime/tz.cpp#L265
So our code will not find tzdata installed by conda.
Could you open a new issue for tzdata related problem? We can work on this as a separated issue.
@raulcd
raulcd
commented
May 19, 2023•

edited

Loading

Copy link


Copy Markdown

Member
The datetime vendored library seems to have had a change lately: https://github.com/apache/arrow/pull/35612/files

And is quite important for the R release to be packed for 12.0.1. I am wondering whether this issue will have to be fixed for the 12.0.1 release too.

It won't as this test was added on a different ticket that won't be added to 12.0.1 .
edit: strikethrough
@h-vetinarih-vetinari
mentioned this pull request
May 24, 2023

Closed

@h-vetinari
h-vetinari
commented
May 24, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
The tzdata stuff is an arrow 13 problem though. More urgently, in conda-forge/arrow-cpp-feedstock#1053, I currently have a failing test test_total_bytes_allocated on arrow 12 (though not arrow 11 or before), but some commit between arrow 12 and current main seems to have fixed this - any ideas?
As I mentioned, the tzdata thing is much less urgent for us in conda-forge than the issue of that failing tests on the feedstock. I now opened a separate issue so this gets (hopefully...) more traction: #35728
@jorisvandenbossche
jorisvandenbossche
commented
May 24, 2023

Copy link


Copy Markdown

Member
Why does arrow depend on this in the first place? It's the first package I see that doesn't work with tzdata.
Arrow unfortunately can't work with the tz database from tzdata, because the underlying date library we use doesn't support the binary format, see #31472 and linked issues.
In other places we skip tests that requite tz database access on Windows, so that should be done for the new scalar test as well, I assume.
@jorisvandenbossche
jorisvandenbossche
commented
May 24, 2023

Copy link


Copy Markdown

Member
I opened #35735 to skip the test requiring a tz database on windows.
@h-vetinari
h-vetinari
commented
May 24, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Arrow unfortunately can't work with the tz database from tzdata, because the underlying date library we use doesn't support the binary format, see #31472 and linked issues.
I don't mind shipping arrow's vendored WindowsZones.xml in a pyarrow-specific location (i.e. site-packages/pyarrow; we can patch pyarrow to point to the right location). I think it would be better to do that than just fail? timezone-operations are pretty common (unless I misunderstand something), so not supporting that for the conda-forge builds would suck IMO
@h-vetinari
h-vetinari
commented
May 24, 2023

Copy link


Copy Markdown

ContributorAuthor
One issue with vendoring is that this doesn't make it easy to update the version / ensure you always have a recent tz database, also when installing an somewhat older pyarrow version.
Absolutely, that's why it would be nice if this could be handled through a "proper" dependency like tzdata. I haven't dived deep into #31472, but I don't understand why the format that tzdata has couldn't be transformed or otherwise made to work for arrow?
@jorisvandenbossche
jorisvandenbossche
commented
May 24, 2023

Copy link


Copy Markdown

Member
That's probably technically possible to implement such transformation, but someone still needs to do the work for that. A potential easier path forward is to update date.h to work with the compiled database on Windows (HowardHinnant/date#564)
@mattijnmattijn
mentioned this pull request
Jun 2, 2023
 Merged

@h-vetinari
h-vetinari
commented
Jul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
h-vetinari
h-vetinari
commented
Jul 24, 2023
View reviewed changes
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
 - clangdev {{ llvm_version }}
 - llvmdev {{ llvm_version }}
 - cython
 - cython <3
@h-vetinarih-vetinariJul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to wait for #36745 (and backport it 12.0.1), but that PR has not landed as quickly as I hoped.
@github-actions
github-actionsBot
commented
Jul 24, 2023

Copy link


Copy Markdown

Revision: e9e6605
Submitted crossbow builds: ursacomputing/crossbow @ actions-b1a7f0eaed
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r41Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r41Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r41Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r41Azure
conda-osx-x64-cpu-r42Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@github-actionsgithub-actionsBot
added

awaiting committer review
Awaiting committer review
and removed

awaiting review
Awaiting review
labels
Jul 24, 2023
@h-vetinari
h-vetinari
commented
Jul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
Jul 24, 2023

Copy link


Copy Markdown

Revision: 17fc588
Submitted crossbow builds: ursacomputing/crossbow @ actions-dd4f8afbf0
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r41Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r41Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r41Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r41Azure
conda-osx-x64-cpu-r42Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@h-vetinarih-vetinari
mentioned this pull request
Jul 24, 2023
 Merged

@h-vetinari
h-vetinari
commented
Jul 25, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou@pitrou@raulcd@assignUser@jorisvandenbossche

This is green now and matches the state of the feedstocks. There are some more minor improvements on the way (e.g. a few - though not all - of the test skips can be removed), but I can also put those in the next sync PR after arrow 13 is officially out.
@koukou
changed the title
GH-35658: Sync conda recipes with feedstocksGH-35658: [Packaging] Sync conda recipes with feedstocksJul 25, 2023
assignUser
assignUser
approved these changes
Jul 26, 2023
View reviewed changes
@assignUserassignUser
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work, thanks @h-vetinari ! I will merge in a day or so if there are not objections
kou
kou
approved these changes
Jul 26, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Comment threaddev/tasks/conda-recipes/arrow-cpp/build-arrow.sh
 EXTRA_CMAKE_ARGS="${EXTRA_CMAKE_ARGS} -DLLVM_LINK_EXECUTABLE=${BUILD_PREFIX}/bin/llvm-link"
 sed -ie "s;protoc-gen-grpc.*$;protoc-gen-grpc=${BUILD_PREFIX}/bin/grpc_cpp_plugin\";g" ../src/arrow/flight/CMakeLists.txt
 sed -ie 's;"--with-jemalloc-prefix\=je_arrow_";"--with-jemalloc-prefix\=je_arrow_" "--with-lg-page\=14";g' ../cmake_modules/ThirdpartyToolchain.cmake
 sed -ie 's;"--with-jemalloc-prefix\=je_arrow_";"--with-jemalloc-prefix\=je_arrow_" "--with-lg-page\=16";g' ../cmake_modules/ThirdpartyToolchain.cmake
@koukouJul 26, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use -DARROW_JEMALLOC_LG_PAGE=16 instead of rewriting ThirdpartyToolchain.cmake.
@h-vetinarih-vetinariJul 26, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds great, I'll change that on the feedstock and in the next sync
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting committer review
Awaiting committer review
labels
Jul 26, 2023
@kou
kou
merged commit 8503c86
into
apache:mainJul 26, 2023
@koukou
removed
the 
awaiting merge
Awaiting merge
label
Jul 26, 2023
@h-vetinari
h-vetinari
deleted the

conda_update

branch
July 26, 2023 23:43
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Aug 2, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 8503c86.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@h-vetinari@loicalleyne
apacheGH-35658: [Packaging] Sync conda recipes with feedstocks (apach…
c1ea980
…e#35637)
Corresponds to conda-forge/arrow-cpp-feedstock#1053 for pyarrow (one failing test on our infra that needs debugging), as well as the state of the feedstock for r-arrow after conda-forge/r-arrow-feedstock#65
* Closes: apache#35658
Authored-by: H. Vetinari <h.vetinari@gmx.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
@h-vetinarih-vetinari
mentioned this pull request
Jun 12, 2024
 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@raulcdraulcdraulcd approved these changes
@assignUserassignUserassignUser approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[Packaging] Sync with conda-forge feedstocks


5 participants

@h-vetinari@kou@assignUser@raulcd@jorisvandenbossche

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GH-35658: [Packaging] Sync conda recipes with feedstocks by h-vetinari · Pull Request #35637 · apache/arrow · GitHub
Skip to content
GH-35658: [Packaging] Sync conda recipes with feedstocks - #35637
Merged
kou merged 10 commits into
apache:mainfrom 
h-vetinari:conda_update
Jul 26, 2023
Merged
GH-35658: [Packaging] Sync conda recipes with feedstocks#35637
kou merged 10 commits into
apache:mainfrom 
h-vetinari:conda_update
Conversation
@h-vetinari
@h-vetinarih-vetinari
commented
May 17, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Contributor
Corresponds to conda-forge/arrow-cpp-feedstock#1053 for pyarrow (one failing test on our infra that needs debugging), as well as the state of the feedstock for r-arrow after conda-forge/r-arrow-feedstock#65
@github-actions
github-actionsBot
commented
May 17, 2023

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@h-vetinarih-vetinari
mentioned this pull request
May 17, 2023
 Merged

@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
May 17, 2023
@h-vetinari
h-vetinari
commented
May 17, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
May 17, 2023

Copy link


Copy Markdown

Revision: 8ef347b
Submitted crossbow builds: ursacomputing/crossbow @ actions-ca516ae62e
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r41Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r41Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r41Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r41Azure
conda-osx-x64-cpu-r42Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@h-vetinari
h-vetinari
commented
May 17, 2023

Copy link


Copy Markdown

ContributorAuthor
Interestingly, the failure of test_total_bytes_allocated on linux from conda-forge/arrow-cpp-feedstock#1053 is not present here. But a weird error on windows:
E pyarrow.lib.ArrowInvalid: Cannot locate timezone 'UTC': Timezone database not found at "C:\Users\VssAdministrator\Downloads\tzdata"

@kou
kou
commented
May 17, 2023

Copy link


Copy Markdown

Member
I don't know why we didn't need timezone database for Windows jobs before but we have a shell script to download timezone database: https://github.com/apache/arrow/blob/main/ci/scripts/download_tz_database.sh
We're using the script on Windows too.
@h-vetinarih-vetinari
changed the title
Sync conda recipes with feedstocksGH-35658: Sync conda recipes with feedstocksMay 18, 2023
@github-actions
github-actionsBot
commented
May 18, 2023

Copy link


Copy Markdown

@github-actions
github-actionsBot
commented
May 18, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #35658has been automatically assigned in GitHub to PR creator.
@h-vetinari
h-vetinari
commented
May 18, 2023

Copy link


Copy Markdown

ContributorAuthor
I don't know why we didn't need timezone database for Windows jobs before but we have a shell script to download timezone database:
Sounds like we should just add a dependency on the tzdata conda-forge package?
@assignUser
assignUser
commented
May 18, 2023

Copy link


Copy Markdown

Member
Sounds like we should just add a dependency on the tzdata conda-forge package?
That sound's good. Thanks for taking care of the sync work, much appreciated!
@kou
kou
commented
May 18, 2023

Copy link


Copy Markdown

Member
It may not work because it seems that tzdata doesn't include WindowsZones.xml.
@h-vetinari
h-vetinari
commented
May 18, 2023

Copy link


Copy Markdown

ContributorAuthor
It may not work because it seems that tzdata doesn't include WindowsZones.xml.
Why does arrow depend on this in the first place? It's the first package I see that doesn't work with tzdata. Also, based on the error message, it seems to be looking in the wrong place? I.e. C:\Users\VssAdministrator\Downloads\tzdata instead of %LIBRARY_PREFIX% (aside from the fact that inndeed, our tzdata does not containWindowsZones.xml).
The tzdata stuff is an arrow 13 problem though. More urgently, in conda-forge/arrow-cpp-feedstock#1053, I currently have a failing test test_total_bytes_allocated on arrow 12 (though not arrow 11 or before), but some commit between arrow 12 and current main seems to have fixed this - any ideas?
@raulcd
raulcd
commented
May 18, 2023•

edited

Loading

Copy link


Copy Markdown

Member
As a note, our nightly builds for wheels on Windows are also failing at the moment with the same error:
 E pyarrow.lib.ArrowInvalid: Cannot locate timezone 'UTC': Timezone database not found at "C:\Users\ContainerAdministrator\Downloads\tzdata"

The first failure is from 10 days ago since this commit was introduced:
053b5ee
@kou
kou
commented
May 19, 2023

Copy link


Copy Markdown

Member
Ah, sorry. I misread this problem.
It seems that WindowsZones.xml isn't related to this problem.
But tzdata path was hard-coded on Windows: https://github.com/apache/arrow/blob/main/cpp/src/arrow/vendored/datetime/tz.cpp#L265
So our code will not find tzdata installed by conda.
Could you open a new issue for tzdata related problem? We can work on this as a separated issue.
@raulcd
raulcd
commented
May 19, 2023•

edited

Loading

Copy link


Copy Markdown

Member
The datetime vendored library seems to have had a change lately: https://github.com/apache/arrow/pull/35612/files

And is quite important for the R release to be packed for 12.0.1. I am wondering whether this issue will have to be fixed for the 12.0.1 release too.

It won't as this test was added on a different ticket that won't be added to 12.0.1 .
edit: strikethrough
@h-vetinarih-vetinari
mentioned this pull request
May 24, 2023

Closed

@h-vetinari
h-vetinari
commented
May 24, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
The tzdata stuff is an arrow 13 problem though. More urgently, in conda-forge/arrow-cpp-feedstock#1053, I currently have a failing test test_total_bytes_allocated on arrow 12 (though not arrow 11 or before), but some commit between arrow 12 and current main seems to have fixed this - any ideas?
As I mentioned, the tzdata thing is much less urgent for us in conda-forge than the issue of that failing tests on the feedstock. I now opened a separate issue so this gets (hopefully...) more traction: #35728
@jorisvandenbossche
jorisvandenbossche
commented
May 24, 2023

Copy link


Copy Markdown

Member
Why does arrow depend on this in the first place? It's the first package I see that doesn't work with tzdata.
Arrow unfortunately can't work with the tz database from tzdata, because the underlying date library we use doesn't support the binary format, see #31472 and linked issues.
In other places we skip tests that requite tz database access on Windows, so that should be done for the new scalar test as well, I assume.
@jorisvandenbossche
jorisvandenbossche
commented
May 24, 2023

Copy link


Copy Markdown

Member
I opened #35735 to skip the test requiring a tz database on windows.
@h-vetinari
h-vetinari
commented
May 24, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Arrow unfortunately can't work with the tz database from tzdata, because the underlying date library we use doesn't support the binary format, see #31472 and linked issues.
I don't mind shipping arrow's vendored WindowsZones.xml in a pyarrow-specific location (i.e. site-packages/pyarrow; we can patch pyarrow to point to the right location). I think it would be better to do that than just fail? timezone-operations are pretty common (unless I misunderstand something), so not supporting that for the conda-forge builds would suck IMO
@h-vetinari
h-vetinari
commented
May 24, 2023

Copy link


Copy Markdown

ContributorAuthor
One issue with vendoring is that this doesn't make it easy to update the version / ensure you always have a recent tz database, also when installing an somewhat older pyarrow version.
Absolutely, that's why it would be nice if this could be handled through a "proper" dependency like tzdata. I haven't dived deep into #31472, but I don't understand why the format that tzdata has couldn't be transformed or otherwise made to work for arrow?
@jorisvandenbossche
jorisvandenbossche
commented
May 24, 2023

Copy link


Copy Markdown

Member
That's probably technically possible to implement such transformation, but someone still needs to do the work for that. A potential easier path forward is to update date.h to work with the compiled database on Windows (HowardHinnant/date#564)
@mattijnmattijn
mentioned this pull request
Jun 2, 2023
 Merged

@h-vetinari
h-vetinari
commented
Jul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
h-vetinari
h-vetinari
commented
Jul 24, 2023
View reviewed changes
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
 - clangdev {{ llvm_version }}
 - llvmdev {{ llvm_version }}
 - cython
 - cython <3
@h-vetinarih-vetinariJul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to wait for #36745 (and backport it 12.0.1), but that PR has not landed as quickly as I hoped.
@github-actions
github-actionsBot
commented
Jul 24, 2023

Copy link


Copy Markdown

Revision: e9e6605
Submitted crossbow builds: ursacomputing/crossbow @ actions-b1a7f0eaed
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r41Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r41Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r41Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r41Azure
conda-osx-x64-cpu-r42Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@github-actionsgithub-actionsBot
added

awaiting committer review
Awaiting committer review
and removed

awaiting review
Awaiting review
labels
Jul 24, 2023
@h-vetinari
h-vetinari
commented
Jul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
Jul 24, 2023

Copy link


Copy Markdown

Revision: 17fc588
Submitted crossbow builds: ursacomputing/crossbow @ actions-dd4f8afbf0
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r41Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r41Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r41Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r41Azure
conda-osx-x64-cpu-r42Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@h-vetinarih-vetinari
mentioned this pull request
Jul 24, 2023
 Merged

@h-vetinari
h-vetinari
commented
Jul 25, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou@pitrou@raulcd@assignUser@jorisvandenbossche

This is green now and matches the state of the feedstocks. There are some more minor improvements on the way (e.g. a few - though not all - of the test skips can be removed), but I can also put those in the next sync PR after arrow 13 is officially out.
@koukou
changed the title
GH-35658: Sync conda recipes with feedstocksGH-35658: [Packaging] Sync conda recipes with feedstocksJul 25, 2023
assignUser
assignUser
approved these changes
Jul 26, 2023
View reviewed changes
@assignUserassignUser
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work, thanks @h-vetinari ! I will merge in a day or so if there are not objections
kou
kou
approved these changes
Jul 26, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Comment threaddev/tasks/conda-recipes/arrow-cpp/build-arrow.sh
 EXTRA_CMAKE_ARGS="${EXTRA_CMAKE_ARGS} -DLLVM_LINK_EXECUTABLE=${BUILD_PREFIX}/bin/llvm-link"
 sed -ie "s;protoc-gen-grpc.*$;protoc-gen-grpc=${BUILD_PREFIX}/bin/grpc_cpp_plugin\";g" ../src/arrow/flight/CMakeLists.txt
 sed -ie 's;"--with-jemalloc-prefix\=je_arrow_";"--with-jemalloc-prefix\=je_arrow_" "--with-lg-page\=14";g' ../cmake_modules/ThirdpartyToolchain.cmake
 sed -ie 's;"--with-jemalloc-prefix\=je_arrow_";"--with-jemalloc-prefix\=je_arrow_" "--with-lg-page\=16";g' ../cmake_modules/ThirdpartyToolchain.cmake
@koukouJul 26, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use -DARROW_JEMALLOC_LG_PAGE=16 instead of rewriting ThirdpartyToolchain.cmake.
@h-vetinarih-vetinariJul 26, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds great, I'll change that on the feedstock and in the next sync
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting committer review
Awaiting committer review
labels
Jul 26, 2023
@kou
kou
merged commit 8503c86
into
apache:mainJul 26, 2023
@koukou
removed
the 
awaiting merge
Awaiting merge
label
Jul 26, 2023
@h-vetinari
h-vetinari
deleted the

conda_update

branch
July 26, 2023 23:43
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Aug 2, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 8503c86.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@h-vetinari@loicalleyne
apacheGH-35658: [Packaging] Sync conda recipes with feedstocks (apach…
c1ea980
…e#35637)
Corresponds to conda-forge/arrow-cpp-feedstock#1053 for pyarrow (one failing test on our infra that needs debugging), as well as the state of the feedstock for r-arrow after conda-forge/r-arrow-feedstock#65
* Closes: apache#35658
Authored-by: H. Vetinari <h.vetinari@gmx.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
@h-vetinarih-vetinari
mentioned this pull request
Jun 12, 2024
 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@raulcdraulcdraulcd approved these changes
@assignUserassignUserassignUser approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[Packaging] Sync with conda-forge feedstocks


5 participants

@h-vetinari@kou@assignUser@raulcd@jorisvandenbossche

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
GH-35658: [Packaging] Sync conda recipes with feedstocks by h-vetinari · Pull Request #35637 · apache/arrow · GitHub
Skip to content
GH-35658: [Packaging] Sync conda recipes with feedstocks - #35637
Merged
kou merged 10 commits into
apache:mainfrom 
h-vetinari:conda_update
Jul 26, 2023
Merged
GH-35658: [Packaging] Sync conda recipes with feedstocks#35637
kou merged 10 commits into
apache:mainfrom 
h-vetinari:conda_update
Conversation
@h-vetinari
@h-vetinarih-vetinari
commented
May 17, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Contributor
Corresponds to conda-forge/arrow-cpp-feedstock#1053 for pyarrow (one failing test on our infra that needs debugging), as well as the state of the feedstock for r-arrow after conda-forge/r-arrow-feedstock#65
@github-actions
github-actionsBot
commented
May 17, 2023

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@h-vetinarih-vetinari
mentioned this pull request
May 17, 2023
 Merged

@github-actionsgithub-actionsBot
added
the 
awaiting review
Awaiting review
label
May 17, 2023
@h-vetinari
h-vetinari
commented
May 17, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
May 17, 2023

Copy link


Copy Markdown

Revision: 8ef347b
Submitted crossbow builds: ursacomputing/crossbow @ actions-ca516ae62e
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r41Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r41Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r41Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r41Azure
conda-osx-x64-cpu-r42Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@h-vetinari
h-vetinari
commented
May 17, 2023

Copy link


Copy Markdown

ContributorAuthor
Interestingly, the failure of test_total_bytes_allocated on linux from conda-forge/arrow-cpp-feedstock#1053 is not present here. But a weird error on windows:
E pyarrow.lib.ArrowInvalid: Cannot locate timezone 'UTC': Timezone database not found at "C:\Users\VssAdministrator\Downloads\tzdata"

@kou
kou
commented
May 17, 2023

Copy link


Copy Markdown

Member
I don't know why we didn't need timezone database for Windows jobs before but we have a shell script to download timezone database: https://github.com/apache/arrow/blob/main/ci/scripts/download_tz_database.sh
We're using the script on Windows too.
@h-vetinarih-vetinari
changed the title
Sync conda recipes with feedstocksGH-35658: Sync conda recipes with feedstocksMay 18, 2023
@github-actions
github-actionsBot
commented
May 18, 2023

Copy link


Copy Markdown

@github-actions
github-actionsBot
commented
May 18, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #35658has been automatically assigned in GitHub to PR creator.
@h-vetinari
h-vetinari
commented
May 18, 2023

Copy link


Copy Markdown

ContributorAuthor
I don't know why we didn't need timezone database for Windows jobs before but we have a shell script to download timezone database:
Sounds like we should just add a dependency on the tzdata conda-forge package?
@assignUser
assignUser
commented
May 18, 2023

Copy link


Copy Markdown

Member
Sounds like we should just add a dependency on the tzdata conda-forge package?
That sound's good. Thanks for taking care of the sync work, much appreciated!
@kou
kou
commented
May 18, 2023

Copy link


Copy Markdown

Member
It may not work because it seems that tzdata doesn't include WindowsZones.xml.
@h-vetinari
h-vetinari
commented
May 18, 2023

Copy link


Copy Markdown

ContributorAuthor
It may not work because it seems that tzdata doesn't include WindowsZones.xml.
Why does arrow depend on this in the first place? It's the first package I see that doesn't work with tzdata. Also, based on the error message, it seems to be looking in the wrong place? I.e. C:\Users\VssAdministrator\Downloads\tzdata instead of %LIBRARY_PREFIX% (aside from the fact that inndeed, our tzdata does not containWindowsZones.xml).
The tzdata stuff is an arrow 13 problem though. More urgently, in conda-forge/arrow-cpp-feedstock#1053, I currently have a failing test test_total_bytes_allocated on arrow 12 (though not arrow 11 or before), but some commit between arrow 12 and current main seems to have fixed this - any ideas?
@raulcd
raulcd
commented
May 18, 2023•

edited

Loading

Copy link


Copy Markdown

Member
As a note, our nightly builds for wheels on Windows are also failing at the moment with the same error:
 E pyarrow.lib.ArrowInvalid: Cannot locate timezone 'UTC': Timezone database not found at "C:\Users\ContainerAdministrator\Downloads\tzdata"

The first failure is from 10 days ago since this commit was introduced:
053b5ee
@kou
kou
commented
May 19, 2023

Copy link


Copy Markdown

Member
Ah, sorry. I misread this problem.
It seems that WindowsZones.xml isn't related to this problem.
But tzdata path was hard-coded on Windows: https://github.com/apache/arrow/blob/main/cpp/src/arrow/vendored/datetime/tz.cpp#L265
So our code will not find tzdata installed by conda.
Could you open a new issue for tzdata related problem? We can work on this as a separated issue.
@raulcd
raulcd
commented
May 19, 2023•

edited

Loading

Copy link


Copy Markdown

Member
The datetime vendored library seems to have had a change lately: https://github.com/apache/arrow/pull/35612/files

And is quite important for the R release to be packed for 12.0.1. I am wondering whether this issue will have to be fixed for the 12.0.1 release too.

It won't as this test was added on a different ticket that won't be added to 12.0.1 .
edit: strikethrough
@h-vetinarih-vetinari
mentioned this pull request
May 24, 2023

Closed

@h-vetinari
h-vetinari
commented
May 24, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
The tzdata stuff is an arrow 13 problem though. More urgently, in conda-forge/arrow-cpp-feedstock#1053, I currently have a failing test test_total_bytes_allocated on arrow 12 (though not arrow 11 or before), but some commit between arrow 12 and current main seems to have fixed this - any ideas?
As I mentioned, the tzdata thing is much less urgent for us in conda-forge than the issue of that failing tests on the feedstock. I now opened a separate issue so this gets (hopefully...) more traction: #35728
@jorisvandenbossche
jorisvandenbossche
commented
May 24, 2023

Copy link


Copy Markdown

Member
Why does arrow depend on this in the first place? It's the first package I see that doesn't work with tzdata.
Arrow unfortunately can't work with the tz database from tzdata, because the underlying date library we use doesn't support the binary format, see #31472 and linked issues.
In other places we skip tests that requite tz database access on Windows, so that should be done for the new scalar test as well, I assume.
@jorisvandenbossche
jorisvandenbossche
commented
May 24, 2023

Copy link


Copy Markdown

Member
I opened #35735 to skip the test requiring a tz database on windows.
@h-vetinari
h-vetinari
commented
May 24, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Arrow unfortunately can't work with the tz database from tzdata, because the underlying date library we use doesn't support the binary format, see #31472 and linked issues.
I don't mind shipping arrow's vendored WindowsZones.xml in a pyarrow-specific location (i.e. site-packages/pyarrow; we can patch pyarrow to point to the right location). I think it would be better to do that than just fail? timezone-operations are pretty common (unless I misunderstand something), so not supporting that for the conda-forge builds would suck IMO
@h-vetinari
h-vetinari
commented
May 24, 2023

Copy link


Copy Markdown

ContributorAuthor
One issue with vendoring is that this doesn't make it easy to update the version / ensure you always have a recent tz database, also when installing an somewhat older pyarrow version.
Absolutely, that's why it would be nice if this could be handled through a "proper" dependency like tzdata. I haven't dived deep into #31472, but I don't understand why the format that tzdata has couldn't be transformed or otherwise made to work for arrow?
@jorisvandenbossche
jorisvandenbossche
commented
May 24, 2023

Copy link


Copy Markdown

Member
That's probably technically possible to implement such transformation, but someone still needs to do the work for that. A potential easier path forward is to update date.h to work with the compiled database on Windows (HowardHinnant/date#564)
@mattijnmattijn
mentioned this pull request
Jun 2, 2023
 Merged

@h-vetinari
h-vetinari
commented
Jul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
h-vetinari
h-vetinari
commented
Jul 24, 2023
View reviewed changes
Comment threaddev/tasks/conda-recipes/arrow-cpp/meta.yaml
 - clangdev {{ llvm_version }}
 - llvmdev {{ llvm_version }}
 - cython
 - cython <3
@h-vetinarih-vetinariJul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to wait for #36745 (and backport it 12.0.1), but that PR has not landed as quickly as I hoped.
@github-actions
github-actionsBot
commented
Jul 24, 2023

Copy link


Copy Markdown

Revision: e9e6605
Submitted crossbow builds: ursacomputing/crossbow @ actions-b1a7f0eaed
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r41Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r41Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r41Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r41Azure
conda-osx-x64-cpu-r42Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@github-actionsgithub-actionsBot
added

awaiting committer review
Awaiting committer review
and removed

awaiting review
Awaiting review
labels
Jul 24, 2023
@h-vetinari
h-vetinari
commented
Jul 24, 2023

Copy link


Copy Markdown

ContributorAuthor
@github-actions crossbow submit -g conda
@github-actions
github-actionsBot
commented
Jul 24, 2023

Copy link


Copy Markdown

Revision: 17fc588
Submitted crossbow builds: ursacomputing/crossbow @ actions-dd4f8afbf0
TaskStatus
conda-cleanAzure
conda-linux-aarch64-cpu-py3Azure
conda-linux-aarch64-cpu-r41Azure
conda-linux-aarch64-cpu-r42Azure
conda-linux-aarch64-cuda-py3Azure
conda-linux-ppc64le-cpu-py3Azure
conda-linux-ppc64le-cuda-py3Azure
conda-linux-x64-cpu-py3Azure
conda-linux-x64-cpu-r41Azure
conda-linux-x64-cpu-r42Azure
conda-linux-x64-cuda-py3Azure
conda-osx-arm64-cpu-py3Azure
conda-osx-arm64-cpu-r41Azure
conda-osx-arm64-cpu-r42Azure
conda-osx-x64-cpu-py3Azure
conda-osx-x64-cpu-r41Azure
conda-osx-x64-cpu-r42Azure
conda-win-x64-cpu-py3Azure
conda-win-x64-cpu-r41Azure
conda-win-x64-cuda-py3Azure
@h-vetinarih-vetinari
mentioned this pull request
Jul 24, 2023
 Merged

@h-vetinari
h-vetinari
commented
Jul 25, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@kou@pitrou@raulcd@assignUser@jorisvandenbossche

This is green now and matches the state of the feedstocks. There are some more minor improvements on the way (e.g. a few - though not all - of the test skips can be removed), but I can also put those in the next sync PR after arrow 13 is officially out.
@koukou
changed the title
GH-35658: Sync conda recipes with feedstocksGH-35658: [Packaging] Sync conda recipes with feedstocksJul 25, 2023
assignUser
assignUser
approved these changes
Jul 26, 2023
View reviewed changes
@assignUserassignUser
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work, thanks @h-vetinari ! I will merge in a day or so if there are not objections
kou
kou
approved these changes
Jul 26, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Comment threaddev/tasks/conda-recipes/arrow-cpp/build-arrow.sh
 EXTRA_CMAKE_ARGS="${EXTRA_CMAKE_ARGS} -DLLVM_LINK_EXECUTABLE=${BUILD_PREFIX}/bin/llvm-link"
 sed -ie "s;protoc-gen-grpc.*$;protoc-gen-grpc=${BUILD_PREFIX}/bin/grpc_cpp_plugin\";g" ../src/arrow/flight/CMakeLists.txt
 sed -ie 's;"--with-jemalloc-prefix\=je_arrow_";"--with-jemalloc-prefix\=je_arrow_" "--with-lg-page\=14";g' ../cmake_modules/ThirdpartyToolchain.cmake
 sed -ie 's;"--with-jemalloc-prefix\=je_arrow_";"--with-jemalloc-prefix\=je_arrow_" "--with-lg-page\=16";g' ../cmake_modules/ThirdpartyToolchain.cmake
@koukouJul 26, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use -DARROW_JEMALLOC_LG_PAGE=16 instead of rewriting ThirdpartyToolchain.cmake.
@h-vetinarih-vetinariJul 26, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds great, I'll change that on the feedstock and in the next sync
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting committer review
Awaiting committer review
labels
Jul 26, 2023
@kou
kou
merged commit 8503c86
into
apache:mainJul 26, 2023
@koukou
removed
the 
awaiting merge
Awaiting merge
label
Jul 26, 2023
@h-vetinari
h-vetinari
deleted the

conda_update

branch
July 26, 2023 23:43
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Aug 2, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 8503c86.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@h-vetinari@loicalleyne
apacheGH-35658: [Packaging] Sync conda recipes with feedstocks (apach…
c1ea980
…e#35637)
Corresponds to conda-forge/arrow-cpp-feedstock#1053 for pyarrow (one failing test on our infra that needs debugging), as well as the state of the feedstock for r-arrow after conda-forge/r-arrow-feedstock#65
* Closes: apache#35658
Authored-by: H. Vetinari <h.vetinari@gmx.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
@h-vetinarih-vetinari
mentioned this pull request
Jun 12, 2024
 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@raulcdraulcdraulcd approved these changes
@assignUserassignUserassignUser approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[Packaging] Sync with conda-forge feedstocks


5 participants

@h-vetinari@kou@assignUser@raulcd@jorisvandenbossche