ORC-1717: Add geometry and geography types by wgtmac · Pull Request #18 · apache/orc-format

wgtmac · 2024-05-11T08:43:24Z

What changes were proposed in this pull request?

Add geometry and geography types to Apache ORC.

Why are the changes needed?

Geospatial support is a missing feature and it is supported by many popular databases, query engines, computing frameworks, etc.

How was this patch tested?

N/A

dongjoon-hyun · 2024-07-19T05:07:32Z

Thank you, @wgtmac . Is it for Apache ORC 1.1.0? If this is final, I'm +1.

dongjoon-hyun · 2024-07-19T05:08:44Z

Also, cc @williamhyun

wgtmac · 2024-07-19T05:45:56Z

@dongjoon-hyun This is not finalized yet. I will update it once ready.

dongjoon-hyun · 2024-07-19T23:26:34Z

Thank you! If it's finalized, please send a head-up email to dev@orc once more.

dongjoon-hyun · 2024-08-22T06:21:20Z

Thank you for updating. Is this ready, @wgtmac ?

wgtmac · 2024-08-22T06:23:56Z

Yes, I believe so. And I will start PoC implementations later.

dongjoon-hyun · 2024-08-22T06:25:50Z

Got it. When do you want to release orc-format v1.1?

Now to help the PoC?
After finishing PoC?

wgtmac · 2024-08-22T06:27:27Z

I think we can wait until both Java and C++ PoC implementations are finished.

dongjoon-hyun · 2024-08-22T15:31:42Z

+1 for the decision. Thank you for the confirmation.

dongjoon-hyun · 2025-04-04T18:22:08Z

Thank you. Is this ready, @wgtmac ?

wgtmac · 2025-04-07T04:00:29Z

Yes, this is ready for review @dongjoon-hyun

cc @ffacs

dongjoon-hyun · 2025-04-08T02:07:44Z

 optional TimestampStatistics timestampStatistics = 9;
 optional bool hasNull = 10;
+ optional uint64 bytes_on_disk = 11;
+ optional CollectionStatistics collection_statistics = 12;


Can we handle these two above lines independently because this is irrelevant to the geometry, @wgtmac ?

I've created #22 to address this.

dongjoon-hyun · 2025-04-08T02:09:53Z

+// Statistics specific to Geometry or Geography type
+message GeospatialStatistics {
+ // A bounding box of geospatial instances
+ optional BoundingBox bbox = 1;


Is bbox a well-known name? I'm just wondering if we can use bounding_box like the other field names.

Yes, bbox is a well-known geospatial acronym.

dongjoon-hyun

Thank you, @wgtmac .

I have three comments.

bytes_on_disk and collection_statistics (https://github.com/apache/orc-format/pull/18/files#r2032264096)
bbox (https://github.com/apache/orc-format/pull/18/files#r2032265481)
This PR is proposed to support Apache Iceberg. However, are you sure that Apache Iceberg community keeps Apache ORC support in the next table format?

If we are not sure about (3), let's revise the PR description by avoiding mentioning Apache Iceberg community.m
I believe this PR itself has enough meaning independently to Apache ORC community. I'm +1 for this additional feature.

cc @williamhyun too

dongjoon-hyun · 2025-04-09T00:33:56Z

Gentle ping, @wgtmac .

wgtmac · 2025-04-09T01:25:37Z

Are you sure that Apache Iceberg community keeps Apache ORC support in the next table format?

I'm not sure about the long-term goal. The recent discussion w.r.t the file format API still considers Apache ORC: https://lists.apache.org/thread/ovyh52m2b6c1hrg4fhw3rx92bzr793n2

dongjoon-hyun · 2025-04-09T03:03:54Z

Thank you, @wgtmac .

I'm referring the latest Apache Iceberg community sync-up meeting on March 16th which superseded the above discussion on February 12th.

https://youtu.be/9BBZKTfcU0s?t=44m10s

wgtmac · 2025-04-09T04:04:27Z

Thanks for the info! I just watched that video and understand the concern from the Iceberg community. There are several issues like missing features (variant, geometry, etc.) and implementation (default values, schema evolution, etc.) For the latter issue, does anyone in the Apache ORC community know the detail?

dongjoon-hyun

+1, LGTM. Thank you for updating, @wgtmac .

dongjoon-hyun · 2025-04-09T23:51:52Z

Since this is an improvement without a breaking change, shall we release as Apache ORC v1.1.0? WDYT, @wgtmac ?

wgtmac · 2025-04-10T01:38:33Z

@dongjoon-hyun Sounds good! I think this can help the implementation a lot.

### What changes were proposed in this pull request? This PR aims to upgrade ORC Format to 1.1.0. ### Why are the changes needed? To bring the latest feature and bug fixes. - https://github.com/apache/orc-format/milestone/2?closed=1 - apache/orc-format#18 - apache/orc-format#19 ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes#2190 from dongjoon-hyun/ORC-1876. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

### What changes were proposed in this pull request? This PR aims to upgrade ORC Format to 1.1.0. ### Why are the changes needed? To bring the latest feature and bug fixes. - https://github.com/apache/orc-format/milestone/2?closed=1 - apache/orc-format#18 - apache/orc-format#19 ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes#2190 from dongjoon-hyun/ORC-1876. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit f5e2413) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

### What changes were proposed in this pull request? This PR aims to upgrade Apache ORC Format to 1.1.0 from 1.1.0 for Apache Spark 4.1.0. ### Why are the changes needed? Apache ORC Format v1.1.0 is released on 2025-04-18. - https://github.com/apache/orc-format/releases/tag/v1.1.0 To bring the latest feature and bug fixes. - https://github.com/apache/orc-format/milestone/2?closed=1 - apache/orc-format#18 - apache/orc-format#19 ### Does this PR introduce _any_ user-facing change? No, there is no behavior change. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes#50588 from dongjoon-hyun/SPARK-51801. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

github-actionsBot added DOCS PROTO labels May 11, 2024

wgtmac marked this pull request as ready for review May 12, 2024 08:15

wgtmac changed the title ~~WIP: Add geometry type~~ORC-1717: Add geometry typeMay 12, 2024

cholmes mentioned this pull request May 24, 2024
Thoughts about a first-class GEOMETRY data type in Parquet? opengeospatial/geoparquet#222
Closed

dongjoon-hyun mentioned this pull request Sep 25, 2024
ORC-1903: Support Geometry and Geography types apache/orc#2032
Closed

wgtmac changed the title ~~ORC-1717: Add geometry type~~ORC-1717: Add geometry and geography typesApr 3, 2025

wgtmac force-pushed the geo branch from d03fe4c to e3d3957Compare April 3, 2025 06:02

dongjoon-hyun reviewed Apr 8, 2025
View reviewed changes

wgtmac added 3 commits April 9, 2025 11:53

WIP: Add geometry type
e25acb9

reflect latest change from parquet proposal
05a09ac

reflect the latest parquet spec
753bffc

wgtmac force-pushed the geo branch from e3d3957 to 753bffcCompare April 9, 2025 03:54

dongjoon-hyun approved these changes Apr 9, 2025
View reviewed changes

dongjoon-hyun merged commit e807a18 into apache:mainApr 9, 2025

dongjoon-hyun added this to the 1.1.0 milestone Apr 9, 2025

This was referenced Apr 15, 2025
[SPARK-51801][BUILD] Upgrade ORC Format to 1.1.0 apache/spark#50588
Closed
ORC-1876: Upgrade to ORC Format 1.1 apache/orc#2190
Closed

dongjoon-hyun mentioned this pull request May 15, 2025
Add support for geospatial types (geometry and geography) apache/orc#2224
Closed

Uh oh!

Conversation

wgtmac commented May 11, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

How was this patch tested?

Uh oh!

dongjoon-hyun commented Jul 19, 2024

Uh oh!

dongjoon-hyun commented Jul 19, 2024

Uh oh!

wgtmac commented Jul 19, 2024

Uh oh!

dongjoon-hyun commented Jul 19, 2024

Uh oh!

dongjoon-hyun commented Aug 22, 2024

Uh oh!

wgtmac commented Aug 22, 2024

Uh oh!

dongjoon-hyun commented Aug 22, 2024

Uh oh!

wgtmac commented Aug 22, 2024

Uh oh!

dongjoon-hyun commented Aug 22, 2024

Uh oh!

dongjoon-hyun commented Apr 4, 2025

Uh oh!

wgtmac commented Apr 7, 2025

Uh oh!

dongjoon-hyunApr 8, 2025

Choose a reason for hiding this comment

Uh oh!

wgtmacApr 9, 2025

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyunApr 8, 2025

Choose a reason for hiding this comment

Uh oh!

wgtmacApr 9, 2025

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Apr 9, 2025

Uh oh!

wgtmac commented Apr 9, 2025

Uh oh!

dongjoon-hyun commented Apr 9, 2025

Uh oh!

wgtmac commented Apr 9, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Apr 9, 2025

Uh oh!

wgtmac commented Apr 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

wgtmac commented May 11, 2024•
edited
Loading

dongjoon-hyun left a comment •
edited
Loading

wgtmac commented Apr 9, 2025•
edited
Loading