GH-825: Add UUID canonical extension type by jhrotko · Pull Request #903 · apache/arrow-java

jhrotko · 2025-11-03T14:51:38Z

What's Changed

Add Uuid Extension type canonical implementation in Java. The proposed implementation was validated in Dremio engine

Closes#825

lidavidm

See previous PR. It is not sufficient to move the existing files, the "uuid" type used in tests was never meant to implement the canonical type. #824

jhrotko · 2025-11-07T13:59:16Z

@lidavidm could you take another look? Let me know if anything else is missing

bodduv

Thank you for adding the canonical type.

jhrotko · 2025-11-11T12:18:42Z

there are some integration tests failling:

 ################# FAILURES #################
FAILED TEST: custom_metadata Java producing, Go consuming
<class 'RuntimeError'>: Command failed: /root/go/bin/arrow-json-integration-test -arrow /tmp/tmprky9nvsk/020d3b5c_generated_custom_metadata.json_as_file -json /tmp/arrow-integration-ypsmstn1/generated_custom_metadata.json -mode VALIDATE
With output:
--------------
arrow-json: JSON schema:
schema:
fields: 4
- sort_of_pandas: type=int8, nullable
metadata: ["pandas": "{}"]
- lots_of_meta: type=int8, nullable
metadata: ["a": "{}", "b": "{}", "c": "{}", "d": "{}", "..": "{}", "w": "{}", "x": "{}", "y": "{}", "z": "{}"]
- unregistered_extension: type=int8, nullable
metadata: ["ARROW:extension:name": "!nonexistent", "ARROW:extension:metadata": "", "ARROW:integration:allow_unregistered_extension": "true"]
- list_with_odd_values: type=list<item: int32, nullable>, nullable
metadata: ["schema_custom_0": "{}", "schema_custom_1": "{}"]
Arrow schema:
schema:
fields: 4
- sort_of_pandas: type=int8, nullable
metadata: ["pandas": "{}"]
- lots_of_meta: type=int8, nullable
metadata: ["..": "{}", "a": "{}", "b": "{}", "c": "{}", "d": "{}", "w": "{}", "x": "{}", "y": "{}", "z": "{}"]
- unregistered_extension: type=int8, nullable
metadata: ["ARROW:extension:name": "!nonexistent", "ARROW:extension:metadata": "", "ARROW:integration:allow_unregistered_extension": "true"]
- list_with_odd_values: type=list<item: int32, nullable>, nullable
metadata: ["schema_custom_1": "{}", "schema_custom_0": "{}"]
arrow-json: schemas did not match
--------------
FAILED TEST: flight_sql Java producing, Go consuming
<class 'RuntimeError'>: Command failed: /root/go/bin/arrow-flight-integration-client -host localhost -port=43857 -scenario flight_sql
With output:
--------------
gc 1 @3.115s 0%: 0.078+0.59+0.008 ms clock, 0.31+0.37/0.34/0.60+0.032 ms cpu, 3->3->0 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 4 P
panic: expected: schema:
fields: 1
- id: type=int64, nullable
metadata: ["ARROW:FLIGHT:SQL:TABLE_NAME": "test", "ARROW:FLIGHT:SQL:IS_AUTO_INCREMENT": "1", "ARROW:FLIGHT:SQL:IS_CASE_SENSITIVE": "0", "ARROW:FLIGHT:SQL:TYPE_NAME": "type_test", "ARROW:FLIGHT:SQL:SCHEMA_NAME": "schema_test", "ARROW:FLIGHT:SQL:IS_SEARCHABLE": "1", "ARROW:FLIGHT:SQL:CATALOG_NAME": "catalog_test", "ARROW:FLIGHT:SQL:PRECISION": "100", "ARROW:FLIGHT:SQL:REMARKS": "test column"], got: schema:
fields: 1
- id: type=int64, nullable
metadata: ["ARROW:FLIGHT:SQL:PRECISION": "100", "ARROW:FLIGHT:SQL:REMARKS": "test column", "ARROW:FLIGHT:SQL:IS_AUTO_INCREMENT": "1", "ARROW:FLIGHT:SQL:IS_CASE_SENSITIVE": "0", "ARROW:FLIGHT:SQL:SCHEMA_NAME": "schema_test", "ARROW:FLIGHT:SQL:TABLE_NAME": "test", "ARROW:FLIGHT:SQL:IS_SEARCHABLE": "1", "ARROW:FLIGHT:SQL:TYPE_NAME": "type_test", "ARROW:FLIGHT:SQL:CATALOG_NAME": "catalog_test"]
goroutine 1 [running]:
main.main()
/arrow/go/arrow/internal/flight_integration/cmd/arrow-flight-integration-client/main.go:52 +0x277
--------------
FAILED TEST: flight_sql:extension Java producing, Go consuming
<class 'RuntimeError'>: Command failed: /root/go/bin/arrow-flight-integration-client -host localhost -port=35551 -scenario flight_sql:extension
With output:
--------------
panic: expected: schema:
fields: 1
- id: type=int64, nullable
metadata: ["ARROW:FLIGHT:SQL:TABLE_NAME": "test", "ARROW:FLIGHT:SQL:IS_AUTO_INCREMENT": "1", "ARROW:FLIGHT:SQL:IS_CASE_SENSITIVE": "0", "ARROW:FLIGHT:SQL:TYPE_NAME": "type_test", "ARROW:FLIGHT:SQL:SCHEMA_NAME": "schema_test", "ARROW:FLIGHT:SQL:IS_SEARCHABLE": "1", "ARROW:FLIGHT:SQL:CATALOG_NAME": "catalog_test", "ARROW:FLIGHT:SQL:PRECISION": "100", "ARROW:FLIGHT:SQL:REMARKS": "test column"], got: schema:
fields: 1
- id: type=int64, nullable
metadata: ["ARROW:FLIGHT:SQL:PRECISION": "100", "ARROW:FLIGHT:SQL:REMARKS": "test column", "ARROW:FLIGHT:SQL:IS_AUTO_INCREMENT": "1", "ARROW:FLIGHT:SQL:IS_CASE_SENSITIVE": "0", "ARROW:FLIGHT:SQL:SCHEMA_NAME": "schema_test", "ARROW:FLIGHT:SQL:TABLE_NAME": "test", "ARROW:FLIGHT:SQL:IS_SEARCHABLE": "1", "ARROW:FLIGHT:SQL:TYPE_NAME": "type_test", "ARROW:FLIGHT:SQL:CATALOG_NAME": "catalog_test"]
goroutine 1 [running]:
main.main()
/arrow/go/arrow/internal/flight_integration/cmd/arrow-flight-integration-client/main.go:52 +0x277
--------------
FAILED TEST: flight_sql Go producing, Java consuming
<class 'RuntimeError'>: Command failed: java -Dio.netty.tryReflectionSetAccessible=true -Darrow.struct.conflict.policy=CONFLICT_APPEND --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED -XX:-UsePerfData --add-reads=org.apache.arrow.flight.core=ALL-UNNAMED -cp /build/java/flight/flight-integration-tests/target/flight-integration-tests-19.0.0-SNAPSHOT-jar-with-dependencies.jar org.apache.arrow.flight.integration.tests.IntegrationTestClient -port 40707 -scenario flight_sql
With output:
--------------
WARNING: Unknown module: org.apache.arrow.flight.core specified to --add-reads
WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens
SLF4J(W): No SLF4J providers were found.
SLF4J(W): Defaulting to no-operation (NOP) logger implementation
SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
Unknown error
expected: '%!s(bool=true)', got: '%!s(bool=false)'
--------------
FAILED TEST: flight_sql:extension Go producing, Java consuming
<class 'RuntimeError'>: Command failed: java -Dio.netty.tryReflectionSetAccessible=true -Darrow.struct.conflict.policy=CONFLICT_APPEND --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED -XX:-UsePerfData --add-reads=org.apache.arrow.flight.core=ALL-UNNAMED -cp /build/java/flight/flight-integration-tests/target/flight-integration-tests-19.0.0-SNAPSHOT-jar-with-dependencies.jar org.apache.arrow.flight.integration.tests.IntegrationTestClient -port 32823 -scenario flight_sql:extension
With output:
--------------
WARNING: Unknown module: org.apache.arrow.flight.core specified to --add-reads
WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens
SLF4J(W): No SLF4J providers were found.
SLF4J(W): Defaulting to no-operation (NOP) logger implementation
SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
Unknown error
expected: '%!s(bool=true)', got: '%!s(bool=false)'
--------------
FAILED TEST: custom_metadata Java producing, Go consuming
<class 'RuntimeError'>: Go C Data Integration call failed: Schemas are different:
- Json Schema: schema:
fields: 4
- sort_of_pandas: type=int8, nullable
metadata: ["pandas": "{}"]
- lots_of_meta: type=int8, nullable
metadata: ["a": "{}", "b": "{}", "c": "{}", "d": "{}", "..": "{}", "w": "{}", "x": "{}", "y": "{}", "z": "{}"]
- unregistered_extension: type=int8, nullable
metadata: ["ARROW:extension:name": "!nonexistent", "ARROW:extension:metadata": "", "ARROW:integration:allow_unregistered_extension": "true"]
- list_with_odd_values: type=list<item: int32, nullable>, nullable
metadata: ["schema_custom_0": "{}", "schema_custom_1": "{}"]
- Imported Schema: schema:
fields: 4
- sort_of_pandas: type=int8, nullable
metadata: ["pandas": "{}"]
- lots_of_meta: type=int8, nullable
metadata: ["..": "{}", "a": "{}", "b": "{}", "c": "{}", "d": "{}", "w": "{}", "x": "{}", "y": "{}", "z": "{}"]
- unregistered_extension: type=int8, nullable
metadata: ["ARROW:extension:name": "!nonexistent", "ARROW:extension:metadata": "", "ARROW:integration:allow_unregistered_extension": "true"]
- list_with_odd_values: type=list<item: int32, nullable>, nullable
metadata: ["schema_custom_1": "{}", "schema_custom_0": "{}"]

It seems the metadata elements when it's a map the order is not maintained. I this a known issue? @lidavidm

lidavidm · 2025-11-11T13:04:32Z

It's apache/arrow-go#571

jhrotko · 2025-11-11T16:07:04Z

@lidavidm it was merged, should this be re-run?

jhrotko · 2025-11-12T08:22:51Z

@lidavidm anything else missing?

bodduv

Thank you for addressing the comments.

lidavidm

This has conflicts

lidavidm · 2025-11-19T10:59:25Z

+ /** Registers the UuidType in the extension type registry. */
+ private static final AtomicBoolean registered = new AtomicBoolean(false);
+
+ static {
+ ExtensionTypeRegistry.register(INSTANCE);
+ }
+
+ /** Register the extension type so it can be used globally. */
+ public static void ensureRegistered() {
+ if (!registered.getAndSet(true)) {
+ // The values don't matter, we just need an instance
+ ExtensionTypeRegistry.register(UuidType.INSTANCE);
+ }
+ }


Isn't all of this redundant? If we have the static initializer, then ensureRegistered can just be a no-op. Or if we're going to use the atomic boolean, then we don't need the static initializer.

this is only used in tests similarly to Opaque extension type

There is no static initializer there. There is one here.

Thank you! I added the static implementation following your suggestion

jbonofre · 2025-11-26T05:43:27Z

Can you rebase please ?

lidavidm · 2025-11-26T05:46:49Z

I think these failures are related to upstream and won't go away on rebase.

jbonofre · 2025-11-26T05:57:20Z

I will check as I try to stabilize our CI. It fails way too often and it's an issue in PRs.

jbonofre · 2025-11-28T10:30:34Z

The PR looks good to me. I will merge as soon as CI is green.

jbonofre · 2025-11-28T10:52:07Z

CI failures are not relevant to this PR. I'm going ahead and merging.

## What's Changed The current UUID vector implementation creates new buffer slices when reading values through holders, which has several drawbacks: - Memory overhead: Each slice creates a new ArrowBuf object - Performance impact: Buffer slicing is slower than direct buffer indexing - Inconsistency: Other fixed-width types (like Decimal) use buffer indexing with a `start` offset field ### Proposed Changes 1. Add `start` field to UUID holders to track buffer offsets: - `UuidHolder`: Add `public int start = 0;` - `NullableUuidHolder`: Add `public int start = 0;` 2. Update `UuidVector` to use buffer indexing 3. Update readers and writers ### Related Work - Original UUID extension type implementation: GH-825 (#903) Closes#948

github-actionsBot added the breaking-change label Nov 3, 2025

This comment has been minimized.
Sign in to view

jhrotko force-pushed the GH-825 branch from 595d2ae to 4e8de67Compare November 3, 2025 14:53

lidavidm requested changes Nov 3, 2025
View reviewed changes

jhrotko changed the title ~~GH-825: Move UuidType to arrow.vector from test~~GH-825: Implement Uuid canonical typeNov 7, 2025

jhrotko changed the title ~~GH-825: Implement Uuid canonical type~~GH-825: Add a canonical extension type UUIDNov 7, 2025

jhrotko force-pushed the GH-825 branch from a298920 to ebca624Compare November 7, 2025 13:58

jhrotko marked this pull request as ready for review November 7, 2025 13:58

jhrotko requested review from jbonofre, laurentgo and wgtmac as code owners November 7, 2025 13:58

jhrotko changed the title ~~GH-825: Add a canonical extension type UUID~~GH-825: Add UUID canonical extension typeNov 7, 2025

bodduv reviewed Nov 9, 2025
View reviewed changes

Comment threadvector/src/main/java/org/apache/arrow/vector/extension/UuidType.java
Comment threadvector/src/main/java/org/apache/arrow/vector/extension/UuidType.java

lidavidm reviewed Nov 10, 2025
View reviewed changes

lidavidm added enhancement PRs that add or improve features. and removed breaking-change labels Nov 11, 2025

github-actionsBot added this to the 18.4.0 milestone Nov 11, 2025

bodduv approved these changes Nov 14, 2025
View reviewed changes

jhrotko requested a review from lidavidm November 14, 2025 14:13

lidavidm reviewed Nov 17, 2025
View reviewed changes

jhrotko force-pushed the GH-825 branch from b04af47 to da42227Compare November 17, 2025 13:02

jhrotko requested a review from lidavidm November 17, 2025 13:56

lidavidm reviewed Nov 19, 2025
View reviewed changes

jhrotko requested a review from lidavidm November 25, 2025 10:02

lidavidm approved these changes Nov 26, 2025
View reviewed changes

jbonofre approved these changes Nov 26, 2025
View reviewed changes

jhrotko force-pushed the GH-825 branch from 9b7c583 to 4195376Compare November 26, 2025 09:18

jhrotko requested review from jbonofre and lidavidm November 26, 2025 10:38

jhrotko added 5 commits November 27, 2025 19:20

apacheGH-825: Move UuidType to arrow.vector from test
6e2cbe5

Add canonical implementation
06a4bb2

Address comments
3cf2be7

Remove FixedSizeExtensionType
c6d1eb2

remove ensureRegistered method from UuidType
f86d83d

jhrotko force-pushed the GH-825 branch from 4195376 to f86d83dCompare November 27, 2025 19:20

jbonofre merged commit 82baf18 into apache:mainNov 28, 2025
22 of 25 checks passed

jhrotko deleted the GH-825 branch November 28, 2025 12:30

jhrotko mentioned this pull request Jan 7, 2026
GH-948: Use buffer indexing for UUID vector #949
Merged

Uh oh!

Conversation

jhrotko commented Nov 3, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's Changed

Uh oh!

This comment has been minimized.

lidavidm left a comment

Choose a reason for hiding this comment

Uh oh!

jhrotko commented Nov 7, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bodduv left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jhrotko commented Nov 11, 2025

Uh oh!

lidavidm commented Nov 11, 2025

Uh oh!

jhrotko commented Nov 11, 2025

Uh oh!

jhrotko commented Nov 12, 2025

Uh oh!

bodduv left a comment

Choose a reason for hiding this comment

Uh oh!

lidavidm left a comment

Choose a reason for hiding this comment

Uh oh!

lidavidmNov 19, 2025

Choose a reason for hiding this comment

Uh oh!

jhrotkoNov 19, 2025

Choose a reason for hiding this comment

Uh oh!

lidavidmNov 19, 2025

Choose a reason for hiding this comment

Uh oh!

jhrotkoNov 19, 2025

Choose a reason for hiding this comment

Uh oh!

jbonofre commented Nov 26, 2025

Uh oh!

lidavidm commented Nov 26, 2025

Uh oh!

jbonofre commented Nov 26, 2025

Uh oh!

jbonofre commented Nov 28, 2025

Uh oh!

jbonofre commented Nov 28, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jhrotko commented Nov 3, 2025•
edited
Loading

jhrotko commented Nov 7, 2025•
edited
Loading