Skip to content

ARROW-13855: [C++][Python] Implement C data interface support for extension types - #11071

Closed
pitrou wants to merge 1 commit into
apache:masterfrom
pitrou:ARROW-13855-export-extension
Closed

ARROW-13855: [C++][Python] Implement C data interface support for extension types#11071
pitrou wants to merge 1 commit into
apache:masterfrom
pitrou:ARROW-13855-export-extension

Conversation

@pitrou

Copy link
Copy Markdown
Member

No description provided.

@github-actions

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

@jorgecarleitao

jorgecarleitao commented Sep 2, 2021

Copy link
Copy Markdown
Member

Can't review the c++ in depth, but from what I can read, I took an equivalent approach (jorgecarleitao/arrow2#363), which I am glad of.

To import

  1. read metadata
  2. detect extension from meta
  3. use extension

To export

  1. create meta map with extension name and meta
  2. fill meta map with remaining meta from field
  3. unwrap inner from extension
  4. expose in c data interface

There is a small divergence in that we remove the extension fields from the field's metadata when consuming from IPC and FFI, and automatically convert them to the DataType::Extension variant (and vice-versa on export). AFAI understand this is an implementation detail that does not impact how they communicate.

@pitrou

Copy link
Copy Markdown
MemberAuthor

Ah, we could remove the metadata fields here too (I think we do it in the IPC reader).

@lidavidmlidavidm left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, looks good to me.

@pitrou
pitrou deleted the ARROW-13855-export-extension branch September 3, 2021 15:47
ViniciusSouzaRoque pushed a commit to s1mbi0se/arrow that referenced this pull request Oct 20, 2021
…ension types
Closesapache#11071 from pitrou/ARROW-13855-export-extension
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: David Li <li.davidm96@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@pitrou@jorgecarleitao@lidavidm