GH-32863: [C++][Parquet] Add DELTA_BYTE_ARRAY encoder to Parquet writer by rok · Pull Request #14341 · apache/arrow

rok · 2022-10-07T04:25:43Z

This is to add DELTA_BYTE_ARRAY encoder.

Closes: [C++][Parquet] Add DELTA_BYTE_ARRAY encoder to Parquet writer #32863

github-actions · 2022-10-07T04:26:03Z

https://issues.apache.org/jira/browse/ARROW-17619

rok · 2022-10-07T04:27:12Z

This is not really review ready yet and needs #14191 and #14293 to merge first.

rok · 2023-01-26T12:35:54Z

note: this should support BYTE_ARRAY and FIXED_LEN_BYTE_ARRAY. PARQUET-2231

github-actions · 2023-02-23T14:15:35Z

Closes: [C++][Parquet] Add DELTA_BYTE_ARRAY encoder to Parquet writer #32863

mapleFU · 2023-02-28T05:34:01Z

(Can this patch go ahead now?)

mapleFU

Though this patch is still draft, I run it for fun. Seems that here is some issue. May that helps

Co-authored-by: mwish <1506118561@qq.com> Co-authored-by: Gang Wu <ustcwg@gmail.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>

Co-authored-by: mwish <1506118561@qq.com>

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

…tor APIs

pitrou · 2023-08-21T17:02:18Z

I'm going to merge this now.

mapleFU · 2023-08-21T17:12:13Z

Thats a long time, bravo!

rok · 2023-08-21T19:25:04Z

Thanks all for helping this along! I'm very happy we got this in!

anjakefala · 2023-08-21T19:51:46Z

Congratulations, @rok!!!

AlenkaF · 2023-08-22T04:21:38Z

Great work everybody, congrats!!

conbench-apache-arrow · 2023-08-22T04:26:14Z

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 94bd0d2.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.

jorisvandenbossche · 2023-08-22T10:00:35Z

We should probably update the python docstring as well:

arrow/python/pyarrow/parquet/core.py

Lines 822 to 827 in fe750ed

column_encoding : string or dict, default None
 Specify the encoding scheme on a per column basis.
 Currently supported values: {'PLAIN', 'BYTE_STREAM_SPLIT'}.
 Certain encodings are only compatible with certain data types.
 Please refer to the encodings section of `Reading and writing Parquet
 files <https://arrow.apache.org/docs/cpp/parquet.html#encodings>`_.

(which was clearly already outdated before this PR as well! ;))

pitrou · 2023-08-22T10:01:24Z

Feel free to open a PR :-)

rok · 2023-08-22T16:13:56Z

I created an issue :) #37312

mapleFU · 2023-11-08T15:00:39Z

- RETURN_NOT_OK(helper.builder->ReserveData(
- std::min<int64_t>(len_, helper.chunk_space_remaining)));


Why previously here ReserveData for min(len_, helper.chunk_space_remaining) here, wouldn't len_ be too large @pitrou

I don't understand: are you commented on the removed code? I'd rather not try to understand code that was removed months ago...

See #38437 for context, where this code is being added back partially

I also don't understand them T_T.
Just confused by this change, so I tried to understand the origin code and find out why this cause the regression, what should I do to fix it

github-actionsBot added Component: Documentation Component: C++ Component: Python Component: Parquet labels Oct 7, 2022

rok force-pushed the ARROW-17619 branch from 5ddafef to 8bdb4b9Compare November 12, 2022 02:16

rok force-pushed the ARROW-17619 branch 2 times, most recently from 37077b0 to 8448815Compare November 25, 2022 15:27

rok force-pushed the ARROW-17619 branch from 8448815 to 6b0761bCompare December 15, 2022 12:12

asfimport mentioned this pull request Jan 5, 2023
[C++][Parquet] Add DELTA_BYTE_ARRAY encoder to Parquet writer #32863
Closed

rok force-pushed the ARROW-17619 branch from 6b0761b to 2cfa3ccCompare February 10, 2023 19:43

rok force-pushed the ARROW-17619 branch from 2cfa3cc to 7f33eccCompare February 23, 2023 14:13

github-actionsBot removed the Component: Documentation label Feb 23, 2023

rok changed the title ~~ARROW-17619: [C++][Parquet] Add DELTA_BYTE_ARRAY encoder to Parquet writer~~GH-32863: [C++][Parquet] Add DELTA_BYTE_ARRAY encoder to Parquet writerFeb 23, 2023

mapleFU reviewed Mar 1, 2023
View reviewed changes

Comment threadcpp/src/parquet/encoding.cc Outdated
Comment threadcpp/src/parquet/encoding.cc Outdated

github-actionsBot added awaiting review Awaiting review awaiting committer review Awaiting committer review awaiting changes Awaiting changes and removed awaiting review Awaiting review awaiting committer review Awaiting committer review labels Mar 1, 2023

rok force-pushed the ARROW-17619 branch from 7f33ecc to 75c2fd3Compare March 1, 2023 17:42

github-actionsBot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Mar 1, 2023

rok force-pushed the ARROW-17619 branch from 6062c17 to b677934Compare March 1, 2023 19:19

github-actionsBot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Mar 1, 2023

rokand others added 16 commits August 21, 2023 16:00

random_byte_array etc
74da1b7

Review feedback
92b457a

Apply suggestions from code review
e4b96aa
Co-authored-by: mwish <1506118561@qq.com> Co-authored-by: Gang Wu <ustcwg@gmail.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>

Review feedback
45a6d51

Review feedback
1e10149

Review feedback
6c6fbde

Review feedback
6930a79

Review feedback
409a6ee

Apply suggestions from code review
7dc32e1
Co-authored-by: mwish <1506118561@qq.com>

Switch to resizable buffer
578d7de

Apply suggestions from code review
8167771
Co-authored-by: Antoine Pitrou <pitrou@free.fr>

Review feedback
0aaa6b4

Review feedback
9f0cdbb

Review feedback
46aa303

Review feedback
acc40ed

Rewrite Binary/FLBA specializations to correctly account for Accumula…
5039cf9
…tor APIs

pitrou approved these changes Aug 21, 2023
View reviewed changes

mapleFU mentioned this pull request Aug 22, 2023
[C++][Parquet] Encoding: Add Benchmark for DELTA_BYTE_ARRAY #37293
Closed

rok mentioned this pull request Aug 22, 2023
[Python][Docs] Update Python docstrings to reflect new parquet encoding option #37312
Closed

jorisvandenbossche mentioned this pull request Oct 24, 2023
[C++] Parquet reading performance regressions #38432
Closed

mapleFU reviewed Nov 8, 2023
View reviewed changes

		RETURN_NOT_OK(helper.builder->ReserveData(
		std::min<int64_t>(len_, helper.chunk_space_remaining)));

Uh oh!

Conversation

rok commented Oct 7, 2022• edited by github-actions BotLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actionsBot commented Oct 7, 2022

Uh oh!

rok commented Oct 7, 2022

Uh oh!

rok commented Jan 26, 2023

Uh oh!

github-actionsBot commented Feb 23, 2023

Uh oh!

mapleFU commented Feb 28, 2023

Uh oh!

mapleFU left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pitrou commented Aug 21, 2023

Uh oh!

mapleFU commented Aug 21, 2023

Uh oh!

rok commented Aug 21, 2023

Uh oh!

anjakefala commented Aug 21, 2023

Uh oh!

AlenkaF commented Aug 22, 2023

Uh oh!

conbench-apache-arrowBot commented Aug 22, 2023

Uh oh!

jorisvandenbossche commented Aug 22, 2023

Uh oh!

pitrou commented Aug 22, 2023

Uh oh!

rok commented Aug 22, 2023

Uh oh!

mapleFUNov 8, 2023

Choose a reason for hiding this comment

Uh oh!

pitrouNov 8, 2023

Choose a reason for hiding this comment

Uh oh!

jorisvandenbosscheNov 8, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mapleFUNov 8, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

rok commented Oct 7, 2022•
edited by github-actions Bot
Loading

jorisvandenbosscheNov 8, 2023•
edited
Loading