Skip to content

PARQUET-247: Add DATE mapping in ValidTypeMap of filter2 - #169

Closed
dongche wants to merge 4 commits into
apache:masterfrom
dongche:PARQUET-247
Closed

dongche wants to merge 4 commits into
apache:masterfrom
dongche:PARQUET-247

Conversation

@dongche

@dongche dongche commented Apr 8, 2015

Copy link
Copy Markdown
Contributor

When Hive use Parquet filter predicate, the Date type is converted to Integer. In ValidTypeMap, it map the class and Parquet type. It throw exception when checking the data type Date.
We should add the map to support Date.

@dongche

dongche commented Apr 10, 2015

Copy link
Copy Markdown
Contributor Author

Hi, @rdblue , @isnotinvain , I am working on HIVE-10253 and find we may need this change, otherwise filter predicate with Date type cannot pass the validation check.

What do you think? Thanks!

@AdamPD

AdamPD commented Apr 29, 2015

Copy link
Copy Markdown

We should be adding the mappings for the other types as well (TIME_MILLIS and TIMESTAMP_MILLIS).

I am having the same challenge doing PPD for Apache Drill with those types. I can certainly create another JIRA item for those, but it seems trivial to add it to this one.

@dongche

dongche commented Apr 29, 2015

Copy link
Copy Markdown
Contributor Author

Sounds good! Thanks.
Updated to add mapping for those types.

@dongche

dongche commented Apr 30, 2015

Copy link
Copy Markdown
Contributor Author

Verified the updates in my local env again. Build and test successfully.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so, for something like this, does it make more sense to use a different java class other than Binary? we could use String for UTF8, and BigNum or something here?

Maybe that could be a higher level API on top of this one though.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add this here is just to make filter checking pass if user needs INT96 for a filter.
It seems INT96 is designed as Binary in PrimitiveType. Maybe higher level API could bind it with other java class. Not sure which to use here other than Binary?

@isnotinvain

Copy link
Copy Markdown
Contributor

per: https://issues.apache.org/jira/browse/PARQUET-201 do you mind adding the rest of the primitives too?

@dongche

dongche commented May 5, 2015

Copy link
Copy Markdown
Contributor Author

Sure, my pleasure!
Updated code with the mapping for OriginalType INT_* and UINT_*

@nandorKollar

Copy link
Copy Markdown
Contributor

Fixed by removing OriginalType check in #219

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants