GH-891: Add ExtensionTypeWriterFactory to TransferPair

[jhrotko]

What's Changed

This PR simplifies extension type writer creation by moving from a factory-based pattern to a type-based pattern. Instead of passing ExtensionTypeWriterFactory instances through multiple API layers, extension types now provide their own writers via a new getNewFieldWriter() method on ArrowType.ExtensionType.

• Added getNewFieldWriter(ValueVector) abstract method to ArrowType.ExtensionType
• Removed ExtensionTypeWriterFactory interface and all implementations
• Removed factory parameters from ComplexCopier, PromotableWriter, and TransferPair APIs
• Updated UnionWriter to support extension types (previously threw UnsupportedOperationException)
• Simplified extension type implementations (UuidType, OpaqueType)

The factory pattern didn't scale well. Each new extension type required creating a separate factory class and passing it through multiple API layers. This was especially painful for external developers who had to maintain two classes per extension type and manage factory parameters everywhere.

The new approach follows the same pattern as MinorType, where each type knows how to create its own writer. This reduces boilerplate, simplifies the API, and makes it easier to implement custom extension types outside arrow-java.

Breaking Changes

• ExtensionTypeWriterFactory has been removed
• Extension types must now implement getNewFieldWriter(ValueVector vector) method
• ExtensionHolders must implement type() which returns the ExtensionType for that Holder
• (Writers are obtained directly from the extension type, not from a factory)

Migration Guide

• Extension types must now implement getNewFieldWriter(ValueVector vector) method

publicclassUuidTypeextendsExtensionType {
...
@OverridepublicFieldWritergetNewFieldWriter(ValueVectorvector) {
returnnewUuidWriterImpl((UuidVector) vector);
}
...
}

• ExtensionHolders must implement type() which returns the ExtensionType for that Holder

publicclassUuidHolderextendsExtensionHolder {
...
@OverridepublicArrowTypetype() {
returnUuidType.INSTANCE;
}

• How to use Extension Writers?
  Before:

  writer.extension(UuidType.INSTANCE);
  writer.addExtensionTypeWriterFactory(extensionTypeWriterFactory);
  writer.writeExtension(value);

  After:

  writer.extension(UuidType.INSTANCE);
  writer.writeExtension(value);

• Also copyAsValue does not need to provide the factory anymore.

Closes#891 .

[jhrotko]

Hello, @lidavidm! Could you take a look at this PR? Also, I don't have permissions to change the label

[jbonofre]

@jhrotko I will take a look on this one as soon as the CI is green (it should be good very soon).

[laurentgo]

I'm not really familiar with arrow vectors to be honest, but I wonder why writers aren't discovered at the same time the extension is being registered as a type? wouldn't that make things simpler from an API/usability perspective?

[jhrotko]

This PR changes how we handle extension type writers in Arrow Java. Instead of using factories that get passed around everywhere, we now let the ArrowType.ExtensionType itself provide the writer implementation. This makes the API simpler and easier to work with, especially if you're implementing custom extension types outside arrow-java.

Problem

In Arrow's type system, each MinorType (INT, FLOAT, VARCHAR, etc.) has its own writer implementation. Extension types are trickier though, because they all share the same MinorType.EXTENSIONTYPE, but each extension type (UUID, Opaque, custom types) needs its own writer implementation. We needed some way to figure out which writer to use for a given extension type.

The previous implementation (commits 34060eb4, 7a7e4edd, 7fe36d70, 8663ffc6) used an ExtensionTypeWriterFactory pattern:

// Usage in ComplexCopierwriter.addExtensionTypeWriterFactory(extensionTypeWriterFactory);
writer.writeExtension(value);

In this pattern, each extension type had a separate factory class (like UuidWriterFactory) that was passed around as a parameter for copy methods. The custom extension writers stored these factories and used them to create the appropriate writer.
The TransferPair interface implementations also needed to carry factories, which polluted other ValueVector classes such as IntVectors and so on.

Why the factory pattern wasn't working well

For developers implementing extension types outside of arrow-java, the situation was even more painful. You had to create and manage two separate classes: one for the type itself (MyCustomType extends ExtensionType) and another for the factory (MyCustomWriterFactory implements ExtensionTypeWriterFactory).

The factory pattern had several issues that made it difficult to scale at this point. Specially if you wanted to use Extension Arrow-java types mixed with out of arrow-java extension types which is something that might happen more often in the future.

The API also got cluttered with factory parameters. Methods like ComplexCopier.copy(reader, writer, extensionTypeWriterFactory), writer.addExtensionTypeWriterFactory(factory), and TransferPair.makeTransferPair(target, factory) all needed these extra parameters. This made the API harder to use and understand.

Finally, the factory pattern created tight coupling between the type definition, the writer implementation, the factory that connects them, and all the code that needs to pass factories around. This made it harder to change any one piece without affecting the others.

The new approach: Let types provide their own writers

I added one abstract method to ArrowType.ExtensionType:

publicabstractclassExtensionTypeextendsArrowType {
// NEW METHODpublicabstractFieldWritergetNewFieldWriter(ValueVectorvector);
// Other methods...
}

publicclassUuidTypeextendsExtensionType {
@OverridepublicFieldWritergetNewFieldWriter(ValueVectorvector) {
returnnewUuidWriterImpl((UuidVector) vector);
}
// Other methods...
}

The new approach is simpler because you only need one class per extension type now, not two. The type knows how to create its own writer. This also means the API is cleaner since there are no more factory parameters cluttering everything. For example, ComplexCopier.copy(reader, writer) and writer.writeExtension(value, type) are much more straightforward, and the type provides the writer internally through extensionType.getNewFieldWriter(vector).

This approach is also consistent with how MinorType already works. The existing pattern for MinorType has each enum constant override getNewFieldWriter() to return its specific writer implementation. Extension types now follow the same pattern:

// MinorType enum (existing pattern)publicenumMinorType {
INT(newInt(...)) {
@OverridepublicFieldWritergetNewFieldWriter(ValueVectorvector) {
returnnewIntWriterImpl((IntVector) vector);
}
},
// ...
}
// ExtensionType (new pattern - same idea)publicclassUuidTypeextendsExtensionType {
@OverridepublicFieldWritergetNewFieldWriter(ValueVectorvector) {
returnnewUuidWriterImpl((UuidVector) vector);
}
}

Finally, there's less coupling overall. Writers don't need to store or manage factories anymore, TransferPair implementations are simpler, and the type information just flows naturally through the ArrowType object.

ComplexCopier got simpler

// OLD: Required factory parametercaseEXTENSIONTYPE:
if (extensionTypeWriterFactory == null) {
thrownewIllegalArgumentException("Must provide ExtensionTypeWriterFactory");
}
if (reader.isSet()) {
Objectvalue = reader.readObject();
if (value != null) {
writer.addExtensionTypeWriterFactory(extensionTypeWriterFactory);
writer.writeExtension(value);
}
}
...
// NEW: Type provides the writercaseEXTENSIONTYPE:
if (reader.isSet()) {
Objectvalue = reader.readObject();
if (value != null) {
writer.writeExtension(value, reader.getField().getType());
}
}
...

[jhrotko]

@lidavidm@xxlaykxx could you also take a look?

[lidavidm]

From a brief glance this approach seems more reasonable

[laurentgo]

(design) should we deprecate this method? (since we now have writeExtension(ExtensionHolder)

[jhrotko]

I am not sure if it should be deprecated, looking at other implementations they usually offer the writeX(X arg) ex.: writeInt, and write(XHolder holder)

[laurentgo]

I believe they do when there's no confusion about type/representation. But here we are relying on lastExtensionType to be set first via getWriter()

[jhrotko]

what is the issue with lastExtensionType state?

[lidavidm]

It does make for a very confusing API; there have been other issues/PRs filed about similar cases. At the very least this must detect and throw an explanatory exception for this case.

Also, it would be good to have an override that lets you supply the extension type so that there's no ambiguity or potential for hard-to-diagnose runtime issues (what if a refactoring in some other part of the code eliminates the getWriter call and now your code is suddenly throwing?). Other types (like decimal) have overrides that let you supply the type, so I think this should too.

[jhrotko]

@lidavidm I am not sure I completely understood your point. I created a commit addressing the lastExtensionType 1a9f205 Let me know if this what you meant

[lidavidm]

Yes, though now I have to question why the type has to be provided twice.

[lidavidm]

Is there not a way to stash the extension type instance?

I believe they do when there's no confusion about type/representation. But here we are relying on lastExtensionType to be set first via getWriter()

It seems even before it implicitly was stashed somehow? Is there not a way we can explicitly stash it? (I'd also be curious why this didn't seem to apply to other parametrized types?)

[xxlaykxx]

Yes, though now I have to question why the type has to be provided twice.

It's for support UnionVector/Writer - because Union could use different vectors inside and it's possible that several impl's of ExtensionType could be used at the same time. From Arrow perspective, they all will be ExtensionVector/Writer - so for the correct determination of the extensionWriter should pass the ArrowType that will be unique

[lidavidm]

Ah...I'd kind of argue that was a weird decision by the union writer (it really shouldn't assume types correspond to type codes (also see #108)), but I guess at this point it's unavoidable, so this API will have to do.

[laurentgo]

This file is part of the 18.3.0 release, so removing it would be a breaking change. We could have a discussion if it okay or not. If it is, maybe we can be a bit more decisive on some other methods (like PromotableWriter#writeExtension(Object)) but otherwise, file need to be kept with a @Deprecated annotation

[jhrotko]

If we decide to move forward with this design it's going to be a breaking change because the factory pattern was completely replaced, not deprecated alongside the new pattern. This will require users to migrate. Fortunately, the migration will be easy: Extension types must implement getNewFieldWriter() method and Extension holders need to implement the type() method as well and remove all factory references. I can provide a better migration guide in the PR description

[lidavidm]

If we're doing a major bump anyways it would be a good chance to improve things.

[jhrotko]

Added migration steps in PR description

[jhrotko]

@laurentgo as you previously suggested I created a thread in mailling dev: https://lists.apache.org/thread/dqfjdvh2owln3gw4tfcmp05rdmqk7hhg

[lidavidm]

@jarohen does XTDB use extension types?

[jarohen]

@jarohen does XTDB use extension types?

@lidavidm it does, but we have our own mechanisms for that outside of arrow-java I'm afraid. We're mostly using arrow-java for the IPC and memory management these days - we needed too many bespoke access patterns of the vectors themselves (particularly DUV) and didn't feel it reasonable to expect you folks to bend over backwards just for us 😄

That said, XT's all open source, feel free to pinch what you like, and I'm more'n happy to talk more about it (maybe a different thread though), if there's anything we can contribute back 🙂

[lidavidm]

Thanks for the confirmation! Just wanted to evaluate how this might affect you if we went ahead, sounds like it wouldn't be a problem

[jhrotko]

@lidavidm I see that the CI failures are unrelated to the changes and other PRs are having the same issues

[lidavidm]

@laurentgo@jbonofre do we want to make this part of the next release (and call it 19.0.0)?

[jbonofre]

@lidavidm yes, I would like to include in the 19.0.0 Arrow Java release (as soon as CI will be green 😄 ).

[lidavidm]

Ok, I assume we'll wait for CI to be fixed, then we can rebase this.

[jbonofre]

CI should be OK now. Thanks @jhrotko for the rebase, I just triggered a build.

[jhrotko]

@lidavidm@jbonofre there was a merge conflict, need to re-run the pipeline

[jhrotko]

@lidavidm@jbonofre can we merge this PR?

[jbonofre]

LGTM

[jhrotko]

@jbonofre@lidavidm@laurentgo@xxlaykxx thank you so much for the reviews and support!