Uh oh!
There was an error while loading. Please reload this page.
ARROW-6883: [C++][Python] Allow writing dictionary deltas - #8811
Closed
pitrou wants to merge 2 commits into
Closed
Conversation
wesm
commented
Dec 3, 2020
Member
Acknowledged, will review when I can |
pitrouforce-pushed
the
ARROW-6883-ipc-write-deltas
branch
from
December 7, 2020 10:41
47a1e91 to
3e11e6bCompare* Add an ipc::IpcWriteOptions member to govern emission of dictionary deltas. If the option is enabled, deltas are detected by checking whether the new dictionary starts with the last emitted one for the same field. However, for nested dictionaries, deltas are not emitted for the outer dictionary, as the read path doesn't support it. * Add a stats() method to ipc::StreamDecoder * Expose the IPC statistics in Python, and add tests
pitrouforce-pushed
the
ARROW-6883-ipc-write-deltas
branch
from
December 10, 2020 11:44
3e11e6b to
16beb03Comparepitrouforce-pushed
the
ARROW-6883-ipc-write-deltas
branch
from
December 10, 2020 11:48
16beb03 to
be9d921Comparepitrou
commented
Dec 14, 2020
MemberAuthor
cc @lidavidm |
lidavidm
commented
Dec 14, 2020
Member
The Flight changes look good to me (though there's still ARROW-10787). |
wesm
approved these changes
Dec 15, 2020
Member
There was a problem hiding this comment.
Might help to be explicit to say that otherwise the order of dictionaries may be altered by a passage from sender to receiver (I assume that's what you mean here by "stream compatibility" but others might be scratching their heads)
MemberAuthor
There was a problem hiding this comment.
No, I simply meant that it's off for compatibility with implementations that don't support delta dictionaries.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add an ipc::IpcWriteOptions member to govern emission of dictionary deltas.
If the option is enabled, deltas are detected by checking whether the new
dictionary starts with the last emitted one for the same field.
However, for nested dictionaries, deltas are not emitted for the outer dictionary,
as the read path doesn't support it.
Add a stats() method to ipc::StreamDecoder
Expose the IPC statistics in Python, and add tests