Skip to content

ARROW-8060: [Python] Make dataset Expression objects serializable - #6702

Closed
kszucs wants to merge 6 commits into
apache:masterfrom
kszucs:ARROW-8060
Closed

ARROW-8060: [Python] Make dataset Expression objects serializable#6702
kszucs wants to merge 6 commits into
apache:masterfrom
kszucs:ARROW-8060

Conversation

@kszucs

Copy link
Copy Markdown
Member

The Scalar are not properly exposed yet and the C++ implementation will change soon so while the serialization will work, we need to refactor this code.

Comment threadpython/pyarrow/_dataset.pyx Outdated
@github-actions

Copy link
Copy Markdown

@jorisvandenbosschejorisvandenbossche left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! added some comments

Comment threadcpp/src/arrow/dataset/filter.h Outdated
Comment threadpython/pyarrow/_dataset.pyx Outdated
Comment threadpython/pyarrow/_dataset.pyx Outdated
Comment threadpython/pyarrow/_dataset.pyx Outdated
Comment threadpython/pyarrow/_dataset.pyx Outdated
Comment threadpython/pyarrow/includes/libarrow_dataset.pxd Outdated
Comment threadpython/pyarrow/_dataset.pyx Outdated
Comment threadpython/pyarrow/_dataset.pyx Outdated
Comment threadpython/pyarrow/_dataset.pyx Outdated
Comment threadcpp/src/arrow/dataset/filter.h Outdated
Comment threadpython/pyarrow/_dataset.pyx Outdated
@wesm

wesm commented Mar 25, 2020

Copy link
Copy Markdown
Member

Is there a JIRA about defining a e.g. Flatbuffers-based serialization protocol (probably better than Protobuf so we avoid having to introduce a library dependency) for expressions? Seems like it might provide a more scalable solution for serialization and could eventually succeed the strategy here

@kszucs

Copy link
Copy Markdown
MemberAuthor

@kszucskszucs left a comment

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, build failure is unrelated.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@kszucs@wesm@jorisvandenbossche@bkietz