Skip to content

GH-38007: [C++] Add VariableShapeTensor implementation - #38008

Merged
rok merged 73 commits into
apache:mainfrom
rok:38007
Feb 24, 2026
Merged

GH-38007: [C++] Add VariableShapeTensor implementation#38008
rok merged 73 commits into
apache:mainfrom
rok:38007

Conversation

@rok

@rokrok commented Oct 4, 2023•
edited
Loading

Copy link
Copy Markdown
Member

Rationale for this change

We want to add VariableShapeTensor extension type definition for arrays containing tensors with variable shapes.

What changes are included in this PR?

This adds a C++ implementation.

Are these changes tested?

Yes.

Are there any user-facing changes?

This adds a new extension type C++.

This was referenced Oct 4, 2023
Closed
Merged
@rokrok mentioned this pull request Oct 11, 2023
Closed
@rok
rokforce-pushed the 38007 branch 2 times, most recently from eec16ec to 6287402CompareOctober 11, 2023 15:47
@rok
rok requested a review from pitrouOctober 12, 2023 11:14
@rokrok assigned rok and unassigned rokOct 12, 2023
@rok
rokforce-pushed the 38007 branch 2 times, most recently from dff5441 to 9c3b464CompareOctober 29, 2023 05:12
@AlenkaFAlenkaF added this to the 15.0.0 milestone Nov 16, 2023
@AlenkaF

AlenkaF commented Nov 16, 2023

Copy link
Copy Markdown
Member

Hi @rok! It would be great to have this in 15.0.0 (added the milestone for visibility). Do you think you have enough bandwidth to work on it at the moment?

@rok

rok commented Nov 16, 2023

Copy link
Copy Markdown
MemberAuthor

@AlenkaF yes I'd very much like to continue working on this, I was waiting for the release before chasing reviewers :D
I'll rebase tonight and would very much appreciate reviews afterwards.

rok
rok commented Nov 23, 2023
View reviewed changes
Comment threadpython/pyarrow/scalar.pxi Outdated
@github-actionsgithub-actionsBot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Nov 23, 2023
pitrou
pitrou reviewed Nov 27, 2023
View reviewed changes
Comment threadpython/pyarrow/scalar.pxi Outdated
@github-actionsgithub-actionsBot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Nov 28, 2023
@rok

rok commented Feb 24, 2026

Copy link
Copy Markdown
MemberAuthor

@pitrou I addressed the review comments and would like to proceed and merge this tomorrow. Any further comments?

pitrou
pitrou reviewed Feb 24, 2026
View reviewed changes

@pitroupitrou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update @rok . A couple more minor comments, otherwise LGTM.

Comment threadcpp/src/arrow/extension/variable_shape_tensor.h Outdated
Comment on lines +94 to +96
const std::vector<int64_t>& permutation = {},
const std::vector<std::string>& dim_names = {},
const std::vector<std::optional<int64_t>>& uniform_shape = {});

@pitroupitrouFeb 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not pass those vectors by value just like in the constructor?

@rokrokFeb 24, 2026

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, 0de39f2

Comment threadcpp/src/arrow/extension/variable_shape_tensor.h Outdated
Comment on lines +109 to +111
const std::vector<int64_t>& permutation = {},
const std::vector<std::string>& dim_names = {},
const std::vector<std::optional<int64_t>>& uniform_shape = {});

@pitroupitrouFeb 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here

@rokrokFeb 24, 2026

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above 0de39f2

Comment threadcpp/src/arrow/extension/tensor_internal.h

#include "arrow/status.h"
#include "arrow/util/print_internal.h"
#include <span>

@pitroupitrouFeb 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please move these with the other stdlib includes above?

@rokrokFeb 24, 2026

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@pitrou

pitrou commented Feb 24, 2026

Copy link
Copy Markdown
Member

By the way, do you want to expose it in Python soon after? That would allow validating against NumPy.

@rok

rok commented Feb 24, 2026

Copy link
Copy Markdown
MemberAuthor

By the way, do you want to expose it in Python soon after? That would allow validating against NumPy.

Yes, I'm jumping on #40354 right after we merge here :)

@rok

rok commented Feb 24, 2026

Copy link
Copy Markdown
MemberAuthor

Oh wait, rebases are cheap now #40354

@rok

rok commented Feb 24, 2026

Copy link
Copy Markdown
MemberAuthor

Thank you for all the reviews @pitrou & others!

@pitrou

pitrou commented Feb 24, 2026

Copy link
Copy Markdown
Member

Thanks for pushing this through @rok ! It's a good thing we finally have an implementation 🎉

@conbench-apache-arrow

conbench-apache-arrowBot commented Feb 25, 2026

Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 2fcc3ec.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 9 possible false positives for unstable benchmarks that are known to sometimes produce them.

@koukou mentioned this pull request Mar 5, 2026
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@AlenkaFAlenkaFAlenkaF left review comments
@pitroupitroupitrou approved these changes
@jorisvandenbosschejorisvandenbosscheAwaiting requested review from jorisvandenbossche
@westonpacewestonpaceAwaiting requested review from westonpace
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees

No one assigned

Labels

Component: C++Component: DocumentationComponent: Python

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[C++][Python] Add VariableShapeTensor implementation

6 participants

@rok@AlenkaF@pitrou@pimdh@jorisvandenbossche@raulcd