Skip to content

GH-41470: [C++] Reuse deduplication logic for direct registration - #41466

Merged
lidavidm merged 3 commits into
apache:mainfrom
bkietz:also-deduplicated-direct-filesystem-registration
May 1, 2024
Merged

GH-41470: [C++] Reuse deduplication logic for direct registration#41466
lidavidm merged 3 commits into
apache:mainfrom
bkietz:also-deduplicated-direct-filesystem-registration

Conversation

@bkietz

@bkietzbkietz commented Apr 30, 2024

Copy link
Copy Markdown
Member

Rationale for this change

As observed in #41309 a crossbow job on mac is failing due to duplicate registration of a factory for the file:// scheme

What changes are included in this PR?

Deduplication of registered filesystem factories is applied to direct registration as well as when merging registries.

Are these changes tested?

No, we just need to verify that the problematic crossbow job is repaired.

Are there any user-facing changes?

No

@bkietzbkietz self-assigned this Apr 30, 2024
@github-actions

Copy link
Copy Markdown

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

@apacheapache deleted a comment from github-actionsBotApr 30, 2024
@apacheapache deleted a comment from github-actionsBotApr 30, 2024
@bkietz

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g homebrew

@github-actions

Copy link
Copy Markdown

Revision: e798267

Submitted crossbow builds: ursacomputing/crossbow @ actions-05f8f5ff1f

TaskStatus
homebrew-cppGitHub Actions

@vibhatha

Copy link
Copy Markdown
Contributor

@github-actions crossbow submit java

@bkietz

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g java-jars

@github-actions

Copy link
Copy Markdown
Invalid group(s) {'java-jars'}. Must be one of {'homebrew', 'verify-rc-wheels', 'conda', 'c-glib', 'verify-rc-source', 'ruby', 'integration', 'wheel', 'fuzz', 'nightly-packaging', 'linux-arm64', 'java', 'vcpkg', 'test', 'verify-rc-source-macos', 'verify-rc-jars', 'packaging', 'nightly-release', 'linux-amd64', 'example-python', 'verify-rc-source-linux', 'go', 'linux', 'verify-rc', 'nightly-tests', 'nightly', 'conan', 'example', 'example-cpp', 'r', 'cpp', 'python', 'verify-rc-binaries'}
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/8897392124

@github-actions

Copy link
Copy Markdown

Revision: e798267

Submitted crossbow builds: ursacomputing/crossbow @ actions-9c4f4373f4

TaskStatus
java-jarsGitHub Actions
verify-rc-source-java-linux-almalinux-8-amd64GitHub Actions
verify-rc-source-java-linux-conda-latest-amd64GitHub Actions
verify-rc-source-java-linux-ubuntu-20.04-amd64GitHub Actions
verify-rc-source-java-linux-ubuntu-22.04-amd64GitHub Actions
verify-rc-source-java-macos-amd64GitHub Actions

@bkietzbkietz changed the title reuse deduplication logic for direct registrationGH-41470: [C++] Reuse deduplication logic for direct registrationApr 30, 2024
@bkietz
bkietz marked this pull request as ready for review April 30, 2024 18:08
@bkietz
bkietz requested a review from lidavidmApril 30, 2024 18:08
@lidavidm

Copy link
Copy Markdown
Member

@github-actions crossbow submit java-jars

matrix:
platform:
- { runs_on: ["macos-latest"], arch: "x86_64"}
- { runs_on: ["macos-13"], arch: "x86_64"}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

macos-latest == macos-14 now so if we want amd64 we have to explicitly request macos-13

@github-actionsgithub-actionsBot added awaiting merge Awaiting merge and removed awaiting committer review Awaiting committer review labels May 1, 2024
@github-actions

Copy link
Copy Markdown

Revision: 2fd3fcc

Submitted crossbow builds: ursacomputing/crossbow @ actions-319edba519

TaskStatus
java-jarsGitHub Actions

@bkietz

Copy link
Copy Markdown
MemberAuthor

The error doesn't seem related to filesystem anymore: https://github.com/ursacomputing/crossbow/actions/runs/8903205653/job/24451424677#step:6:16247

I'm not sure what it means though

@vibhatha

Copy link
Copy Markdown
Contributor

@lidavidm this seems to be from the recent maven build update?

@lidavidm

Copy link
Copy Markdown
Member

We can merge this regardless and @vibhatha can file a new ticket.

@lidavidm
lidavidm merged commit 14c54bb into apache:mainMay 1, 2024
@conbench-apache-arrow

Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 14c54bb.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 22 possible false positives for unstable benchmarks that are known to sometimes produce them.

@bkietz
bkietz deleted the also-deduplicated-direct-filesystem-registration branch May 2, 2024 14:32
@raulcd

Copy link
Copy Markdown
Member

I will have to manually include: 2fd3fcc into 16.1.0 in order to have a passing java-jars job as we don't want to add the other commits of this PR (the filesystems split should go into 17.0.0 instead of 16.1.0)
I plan to add the above as a manual commit along with the fix for: #41578

vibhatha pushed a commit to vibhatha/arrow that referenced this pull request May 25, 2024
…on (apache#41466)
### Rationale for this change
As observed in apache#41309 a crossbow job on mac is failing due to duplicate registration of a factory for the file:// scheme
### What changes are included in this PR?
Deduplication of registered filesystem factories is applied to direct registration as well as when merging registries.
### Are these changes tested?
No, we just need to verify that the problematic crossbow job is repaired.
### Are there any user-facing changes?
No
* GitHub Issue: apache#41470
Lead-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Co-authored-by: David Li <li.davidm96@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@bkietz@vibhatha@lidavidm@raulcd