Skip to content

[Parquet] Add support to read Parquet files written with old 2-level list structures. - #3774

Merged
rdblue merged 10 commits into
apache:masterfrom
SinghAsDev:pq_compat
Feb 1, 2022
Merged

[Parquet] Add support to read Parquet files written with old 2-level list structures.#3774
rdblue merged 10 commits into
apache:masterfrom
SinghAsDev:pq_compat

Conversation

@SinghAsDev

@SinghAsDevSinghAsDev commented Dec 19, 2021•
edited
Loading

Copy link
Copy Markdown
Contributor

Add support to read Parquet files written with old 2-level list structures. This should resolve#3759.

@SinghAsDev

SinghAsDev commented Dec 19, 2021

Copy link
Copy Markdown
ContributorAuthor

@rdblue@RussellSpitzer can you help review this? Thanks

kbendick
kbendick reviewed Dec 21, 2021
View reviewed changes
Comment threadparquet/src/test/java/org/apache/iceberg/parquet/TestParquetSchemaUtil.java Outdated
@jackye1995

jackye1995 commented Dec 22, 2021

Copy link
Copy Markdown
Contributor

3773 is merged, could you rebase?

@SinghAsDev

SinghAsDev commented Dec 28, 2021

Copy link
Copy Markdown
ContributorAuthor

3773 is merged, could you rebase?

@jackye1995 done

@wizardxz

wizardxz commented Jan 4, 2022

Copy link
Copy Markdown

LGTM

@SinghAsDev

SinghAsDev commented Jan 10, 2022

Copy link
Copy Markdown
ContributorAuthor

@jackye1995@RussellSpitzer@kbendick@rdblue can I get a review on this, thanks

RussellSpitzer
RussellSpitzer reviewed Jan 13, 2022
View reviewed changes
Comment threadparquet/src/test/java/org/apache/iceberg/parquet/TestParquetSchemaUtil.java Outdated
RussellSpitzer
RussellSpitzer reviewed Jan 13, 2022
View reviewed changes
Comment threadparquet/src/main/java/org/apache/iceberg/parquet/MessageTypeToType.java Outdated
RussellSpitzer
RussellSpitzer reviewed Jan 13, 2022
View reviewed changes
Comment threadparquet/src/main/java/org/apache/iceberg/parquet/ParquetSchemaUtil.java Outdated
RussellSpitzer
RussellSpitzer reviewed Jan 13, 2022
View reviewed changes
Comment threadparquet/src/main/java/org/apache/iceberg/parquet/MessageTypeToType.java Outdated
@SinghAsDev
Add tests sepcifically for old list written by parquet-avro and parqu…
a84a545
…et-thrift. Also rename ParquetSchemaUtil.isListElementType to ParquetSchemaUtil.isOldListElementType
RussellSpitzer
RussellSpitzer approved these changes Jan 19, 2022
View reviewed changes

@RussellSpitzerRussellSpitzer left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me, but I'm not a parquet expert so @rdblue could you also take a quick look before we merge?

rdblue
rdblue reviewed Jan 21, 2022
View reviewed changes
Comment threadparquet/src/main/java/org/apache/iceberg/parquet/ParquetSchemaUtil.java Outdated
rdblue
rdblue reviewed Jan 21, 2022
View reviewed changes
Comment threadparquet/src/main/java/org/apache/iceberg/parquet/MessageTypeToType.java Outdated
rdblue
rdblue reviewed Jan 21, 2022
View reviewed changes
Comment threadparquet/src/main/java/org/apache/iceberg/parquet/MessageTypeToType.java Outdated
rdblue
rdblue reviewed Jan 21, 2022
View reviewed changes
Comment threadparquet/src/main/java/org/apache/iceberg/parquet/ApplyNameMapping.java
rdblue
rdblue reviewed Jan 21, 2022
View reviewed changes
Comment threadparquet/src/main/java/org/apache/iceberg/parquet/ParquetTypeVisitor.java
rdblue
rdblue reviewed Jan 21, 2022
View reviewed changes
Comment threadparquet/src/main/java/org/apache/iceberg/parquet/PruneColumns.java Outdated
rdblue
rdblue reviewed Jan 22, 2022
View reviewed changes
Comment threadparquet/src/main/java/org/apache/iceberg/parquet/TypeWithSchemaVisitor.java Outdated
rdblue
rdblue reviewed Jan 22, 2022
View reviewed changes
Comment threadspark/v3.2/spark/src/test/java/org/apache/iceberg/spark/data/TestSparkParquetReader.java Outdated
rdblue
rdblue reviewed Jan 22, 2022
View reviewed changes
Comment threadspark/v3.2/spark/src/test/java/org/apache/iceberg/spark/data/TestSparkParquetReader.java Outdated
rdblue
rdblue reviewed Jan 22, 2022
View reviewed changes
Comment threadspark/v3.2/spark/src/test/java/org/apache/iceberg/spark/data/TestSparkParquetReader.java Outdated
rdblue
rdblue reviewed Jan 22, 2022
View reviewed changes
Comment threadspark/v3.2/spark/src/test/java/org/apache/iceberg/spark/data/TestSparkParquetReader.java Outdated
rdblue
rdblue reviewed Jan 25, 2022
View reviewed changes
Comment threadparquet/src/main/java/org/apache/iceberg/parquet/PruneColumns.java Outdated
rdblue
rdblue reviewed Jan 25, 2022
View reviewed changes
Comment threadparquet/src/main/java/org/apache/iceberg/parquet/TypeWithSchemaVisitor.java Outdated
rdblue
rdblue reviewed Jan 25, 2022
View reviewed changes
Comment threadparquet/src/main/java/org/apache/iceberg/parquet/TypeWithSchemaVisitor.java Outdated
rdblue
rdblue reviewed Jan 25, 2022
View reviewed changes
Comment threadparquet/src/main/java/org/apache/iceberg/parquet/TypeWithSchemaVisitor.java Outdated
rdblue
rdblue reviewed Jan 25, 2022
View reviewed changes
Comment threadparquet/src/main/java/org/apache/iceberg/parquet/ParquetTypeVisitor.java Outdated
T elementResult = null;
if (repeatedElement.getFieldCount() > 0) {
Type elementField = repeatedElement.getType(0);
if (repeatedElement.isPrimitive() || repeatedElement.asGroupType().getFieldCount() > 0) {

@rdbluerdblueJan 25, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes to this class are inconsistent with the changes to the TypeWithSchemaVisitor. Here, the repeated element is always visited (beforeRepeatedElement call above) and may be processed again as the element. The other avoids pushing the name on the stack. If beforeRepeatedElement were used to track names, I think it would get a duplicate name for the repeated group.

@SinghAsDevSinghAsDevJan 25, 2022

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, addressed this. It would be nice to have some tests to check this behavior. But, I don't think we need to block on that, unless you disagree.

rdblue
rdblue reviewed Jan 28, 2022
View reviewed changes
Comment threadparquet/src/main/java/org/apache/iceberg/parquet/ParquetSchemaUtil.java Outdated
rdblue
rdblue reviewed Jan 28, 2022
View reviewed changes
Comment threadparquet/src/main/java/org/apache/iceberg/parquet/ParquetTypeVisitor.java
rdblue
rdblue reviewed Jan 28, 2022
View reviewed changes
Comment threadparquet/src/main/java/org/apache/iceberg/parquet/ParquetTypeVisitor.java Outdated
rdblue
rdblue reviewed Jan 28, 2022
View reviewed changes
Comment threadparquet/src/main/java/org/apache/iceberg/parquet/ParquetTypeVisitor.java Outdated
rdblue
rdblue reviewed Jan 28, 2022
View reviewed changes
Comment threadparquet/src/main/java/org/apache/iceberg/parquet/ParquetTypeVisitor.java Outdated
rdblue
rdblue reviewed Jan 28, 2022
View reviewed changes
Comment threadparquet/src/main/java/org/apache/iceberg/parquet/ParquetSchemaUtil.java
rdblue
rdblue reviewed Jan 28, 2022
View reviewed changes
Comment threadparquet/src/main/java/org/apache/iceberg/parquet/TypeWithSchemaVisitor.java Outdated
rdblue
rdblue reviewed Jan 28, 2022
View reviewed changes
Comment threadparquet/src/main/java/org/apache/iceberg/parquet/TypeWithSchemaVisitor.java Outdated
private static <T> T visitTwoLevelList(Types.ListType iListType, Types.NestedField iListElement, GroupType pListType,
Type pListElement, TypeWithSchemaVisitor<T> visitor) {
T elementResult = visitField(iListElement, pListElement, visitor);

@rdbluerdblueJan 28, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style: extra newline. Also, we either wrap argument lists at the same level or start all arguments on the next line at 2 indents.

@SinghAsDevSinghAsDevJan 30, 2022

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the style used in various parts of code are different. For example, IIUC ParquetReadSupport.prepareForRead is different than what you are saying. Earlier you also had mentioned Iceberg does not use new param at new line pattern. Updating this part to keep the same level (align params start with previous line) and wrap.

Let me know which style we should try to follow and I can try to update the intellij-style that we provide with Iceberg repo accordingly. I don't know if it is possible, but I can try.

@rdbluerdblueJan 30, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't worry about where still is incorrect in other places. We'll eventually track those down and fix them, but we do want to keep style from diverging when it is caught by a review. So please do fix this.

@SinghAsDevSinghAsDevJan 30, 2022•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should already be fixed. Does the update still have style issue?

@rdblue

rdblue commented Jan 28, 2022

Copy link
Copy Markdown
Contributor

@SinghAsDev, I did another round of review. This also still needs to update Iceberg generics, I think.

@SinghAsDev

SinghAsDev commented Jan 30, 2022

Copy link
Copy Markdown
ContributorAuthor

@SinghAsDev, I did another round of review. This also still needs to update Iceberg generics, I think.

Thanks for thorough reviews @rdblue ! I thought I already covered the Iceberg generic with this change. Am I missing something?

@rdblue

rdblue commented Jan 30, 2022•
edited
Loading

Copy link
Copy Markdown
Contributor

@SinghAsDev, looks like it is missing a test for Iceberg generics. I see the update for it now. Thanks!

@SinghAsDev
Revert back to previous version of visitThreeLevelList as repeatedEle…
c44c909
…ment needs to be popped out after visiting list.
@rdblue
rdblue merged commit 6644393 into apache:masterFeb 1, 2022
@rdblue

rdblue commented Feb 1, 2022

Copy link
Copy Markdown
Contributor

Thanks, @SinghAsDev!

@SinghAsDev
SinghAsDev deleted the pq_compat branch February 1, 2022 21:26
SinghAsDev added a commit to pinterest/iceberg that referenced this pull request Apr 8, 2022
@SinghAsDev
Parquet: Support reading 2-level lists in imported data files (apache…
161dc99
…#3774)
sunchao pushed a commit to sunchao/iceberg that referenced this pull request May 9, 2023
@SinghAsDev@aokolnychyi
Parquet: Support reading 2-level lists in imported data files (apache…
06c080b
…#3774)
@josephperez3josephperez3 mentioned this pull request Mar 23, 2026
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rdbluerdbluerdblue approved these changes
@RussellSpitzerRussellSpitzerRussellSpitzer approved these changes
+2 more reviewers
@kbendickkbendickkbendick left review comments
@wizardxzwizardxzwizardxz approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

builddataflinkparquetpigspark

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add backwards compatibility read support for Parquet list

6 participants

@SinghAsDev@jackye1995@wizardxz@rdblue@RussellSpitzer@kbendick