ORC-1903: Support `Geometry` and `Geography` types by ffacs · Pull Request #2032 · apache/orc

ffacs · 2024-09-22T16:04:00Z

What changes were proposed in this pull request?

Support Geometry and Geography types for java side

Why are the changes needed?

Add support for Geometry and Geography types

How was this patch tested?

UT passed

Was this patch authored or co-authored using generative AI tooling?

YES

ffacs · 2024-09-23T13:48:11Z

cc @wgtmac @dongjoon-hyun

wgtmac · 2024-09-24T05:10:43Z

Thanks for implementing this! I will hold my review until the spec has been finalized.

dongjoon-hyun · 2024-09-25T05:52:59Z

Thank you, @ffacs .

To @wgtmac , do you mean the ORC format PR? Or, Iceberg document? Otherwise, could you provide some pointers to track this part, the spec has been finalized?

Thanks for implementing this! I will hold my review until the spec has been finalized.

dongjoon-hyun · 2024-09-25T05:55:18Z


 <mockito.version>5.10.0</mockito.version>
- <orc-format.version>1.0.0</orc-format.version>
+ <orc-format.version>1.0.0-SNAPSHOT</orc-format.version>


? AFAIK, we didn't change anything relevant until now, do we need -SNAPSHOT?

Thank you for the review. The spec of geometry type haven't been merged into orc format yet, so I created a private package for developing here. Maybe it's a little bit early for code reviewing, let's waiting for the spec finished first.

dongjoon-hyun · 2024-09-25T05:57:03Z

 <surefire.version>3.0.0-M5</surefire.version>
 <test.tmp.dir>${project.build.directory}/testing-tmp</test.tmp.dir>
 <zstd-jni.version>1.5.6-4</zstd-jni.version>
+ <jts.version>1.19.0</jts.version>


This seems to be too old because it's released on Jun 21, 2022.
Why don't we use the latest one? Is there any incompatibility issue with something?
https://mvnrepository.com/artifact/org.locationtech.jts/jts-core/1.20.0

wgtmac · 2024-09-25T06:06:21Z

apache/parquet-format#240 is still under review and subject to change. I think we can wait until it has been finalized. @dongjoon-hyun

dongjoon-hyun · 2024-09-25T06:35:37Z

Ya, I want to check ETA for this work. Do you think we can have this for Apache ORC 2.1?

wgtmac · 2024-09-25T06:52:10Z

If 2.1.0 will be released around January 17, 2025, then I think yes.

dongjoon-hyun · 2025-01-03T16:39:40Z

Happy New Year, @ffacs and @wgtmac .

We need to release Apache ORC 2.1.0 this month.

Given that the status of Apache Parquet community, it's not finished still, right?

PARQUET-2471: Add GEOMETRY and GEOGRAPHY logical types parquet-format#240

Let me remove this from ORC 2.1.0 milestone for now, @wgtmac .

dongjoon-hyun · 2025-01-03T16:40:12Z

For the record, we can have this at Apache ORC 3.0.0.

wgtmac · 2025-01-05T13:50:00Z

Yes, the discussion of Geometry type on the Iceberg and Parquet side took longer time than expected. I don't think it will be closed soon. Moving it to 3.0.0 seems reasonable.

dongjoon-hyun · 2025-01-05T16:17:34Z

Thank you for the confirmation, @wgtmac .

dongjoon-hyun · 2025-04-10T00:02:33Z

It seems that we are ready to release Apache ORC Format v1.1.0 to support this PR.

https://github.com/apache/orc-format/milestone/2?closed=1

dongjoon-hyun · 2025-04-29T15:41:40Z

Can we resume this since the main branch is using ORC Format v1.1, @ffacs and @wgtmac ?

Apache Iceberg v1.9.0 is also released with geo type.

https://iceberg.apache.org/releases/#190-release
- Spec: Support geo type iceberg#10981

ffacs · 2025-04-30T03:32:43Z

Can we resume this since the main branch is using ORC Format v1.1, @ffacs and @wgtmac ?
Apache Iceberg v1.9.0 is also released with geo type.
https://iceberg.apache.org/releases/#190-release
Spec: Support geo type iceberg#10981

Yes, I'd start work on this these days.

dongjoon-hyun · 2025-04-30T12:59:25Z

Thank you so much!

dongjoon-hyun · 2025-06-02T18:06:26Z

Thank you for updating PR, @ffacs .

dongjoon-hyun · 2025-06-10T15:44:49Z

Thank you for updating, @ffacs .

dongjoon-hyun

+1, LGTM. Thank you so much, @ffacs .

cc @williamhyun , @wgtmac , @cxzl25 , @guiyanakuang

and cc @MrPowers , @mrpowers-wb, @jiayuasu from #2224

cxzl25

LGTM

dongjoon-hyun · 2025-06-12T14:22:10Z

Thank you, @cxzl25 .

williamhyun · 2025-06-12T16:50:54Z

Thank you @ffacs, @dongjoon-hyun, @wgtmac, and @cxzl25!
Will be merging this now.

dongjoon-hyun · 2025-06-12T19:54:56Z

Thank you, @williamhyun .

ffacs · 2025-06-15T10:03:31Z

Thank you @dongjoon-hyun @williamhyun @cxzl25 .

### What changes were proposed in this pull request? This PR aims to upgrade ORC to 2.2.0. 2.2.0 RC1 is currently under voting. ### Why are the changes needed? Apache ORC 2.2.0 is a new feature release. - https://github.com/apache/orc/releases/tag/v2.2.0 - apache/orc#2032 - apache/orc#2338 - apache/orc#2269 - apache/orc#2249 - apache/orc#2144 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes#51676 from williamhyun/ORC-2.2.0. Authored-by: William Hyun <william@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

github-actionsBot added BUILD JAVA labels Sep 22, 2024

dongjoon-hyun marked this pull request as draft September 22, 2024 21:31

cxzl25 reviewed Sep 23, 2024
View reviewed changes

Comment threadjava/core/src/java/org/apache/orc/impl/ColumnStatisticsImpl.java Outdated

cxzl25 reviewed Sep 23, 2024
View reviewed changes

Comment threadjava/core/src/java/org/apache/orc/impl/ColumnStatisticsImpl.java Outdated

dongjoon-hyun reviewed Sep 25, 2024
View reviewed changes

dongjoon-hyun added this to the 2.1.0 milestone Sep 25, 2024

dongjoon-hyun removed this from the 2.1.0 milestone Jan 3, 2025

dongjoon-hyun mentioned this pull request May 15, 2025
Add support for geospatial types (geometry and geography) #2224
Closed

ffacs force-pushed the geo branch from c5fef46 to 293d918Compare June 2, 2025 15:09

ffacs changed the title ~~[WIP] Add support for geometry type~~ORC-1903: Add support for geospatial typesJun 2, 2025

ffacs marked this pull request as ready for review June 2, 2025 15:35

dongjoon-hyun reviewed Jun 3, 2025
View reviewed changes

Comment threadjava/core/src/java/org/apache/orc/TypeDescription.java Outdated

dongjoon-hyun reviewed Jun 3, 2025
View reviewed changes

Comment threadjava/core/src/test/org/apache/orc/TestOrcGeospatial.java Outdated

ffacs added 13 commits June 10, 2025 23:40

Fix spotless
d5cd485

Fix spotbugs
0338510

Fix checkstyle
4b50093

Fix spotless again
daba915

Fix some bugs
e4099f2

Fix style
866bf64

Fix building
f2c1372

Yet another fix
cedf9b6

Refactor
abb8d01

fix indentation
5bac508

revert changes in TestColumnStatisticsImpl.java
0f85db0

Refactor serdes
2219a58

inline updateBounds
79ac6ed

ffacs force-pushed the geo branch from 2cc7784 to 79ac6edCompare June 10, 2025 15:43

dongjoon-hyun reviewed Jun 10, 2025
View reviewed changes

Comment threadjava/core/src/test/org/apache/orc/TestOrcGeospatial.java Outdated

dongjoon-hyun reviewed Jun 10, 2025
View reviewed changes

Comment threadjava/core/src/test/org/apache/orc/TestOrcGeospatial.java Outdated

dongjoon-hyun reviewed Jun 10, 2025
View reviewed changes

Comment threadjava/core/src/test/org/apache/orc/TestOrcGeospatial.java Outdated

support Java 25-ea
95c59a5

dongjoon-hyun approved these changes Jun 11, 2025
View reviewed changes

cxzl25 approved these changes Jun 12, 2025
View reviewed changes

williamhyun approved these changes Jun 12, 2025
View reviewed changes

williamhyun closed this in 1e5c161Jun 13, 2025

dongjoon-hyun mentioned this pull request Jul 27, 2025
[SPARK-52967][BUILD] Upgrade ORC to 2.2.0 apache/spark#51676
Closed

Uh oh!

Conversation

ffacs commented Sep 22, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

Uh oh!

Uh oh!

ffacs commented Sep 23, 2024

Uh oh!

wgtmac commented Sep 24, 2024

Uh oh!

dongjoon-hyun commented Sep 25, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyunSep 25, 2024

Choose a reason for hiding this comment

Uh oh!

ffacsSep 26, 2024

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyunSep 25, 2024

Choose a reason for hiding this comment

Uh oh!

wgtmac commented Sep 25, 2024

Uh oh!

dongjoon-hyun commented Sep 25, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wgtmac commented Sep 25, 2024

Uh oh!

dongjoon-hyun commented Jan 3, 2025

Uh oh!

dongjoon-hyun commented Jan 3, 2025

Uh oh!

wgtmac commented Jan 5, 2025

Uh oh!

dongjoon-hyun commented Jan 5, 2025

Uh oh!

dongjoon-hyun commented Apr 10, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Apr 29, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ffacs commented Apr 30, 2025

Uh oh!

dongjoon-hyun commented Apr 30, 2025

Uh oh!

dongjoon-hyun commented Jun 2, 2025

Uh oh!

Uh oh!

Uh oh!

dongjoon-hyun commented Jun 10, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

cxzl25 left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jun 12, 2025

Uh oh!

williamhyun commented Jun 12, 2025

Uh oh!

dongjoon-hyun commented Jun 12, 2025

Uh oh!

ffacs commented Jun 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

ffacs commented Sep 22, 2024•
edited
Loading

dongjoon-hyun commented Sep 25, 2024•
edited
Loading

dongjoon-hyun commented Sep 25, 2024•
edited
Loading

dongjoon-hyun commented Apr 10, 2025•
edited
Loading

dongjoon-hyun commented Apr 29, 2025•
edited
Loading