[opt](paimon) support read paimon variant type using jni reader by zhangstar333 · Pull Request #66334 · apache/doris

zhangstar333 · 2026-07-31T09:48:43Z

What problem does this PR solve?

Problem Summary:
support read paimon variant type using jni reader

mysql> desc test_paimon_variant.db1.variant_smoke;
+---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------+------+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------+------+---------+-------+
| id | bigint | Yes | true | NULL | |
| payload | variant<PROPERTIES ("variant_max_subcolumns_count" = "0","variant_enable_typed_paths_to_sparse" = "false","variant_max_sparse_column_statistics_size" = "10000","variant_sparse_hash_shard_count" = "1")> | Yes | true | NULL | |
+---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------+------+---------+-------+
2 rows in set (0.01 sec)
mysql> select * from test_paimon_variant.db1.variant_smoke;
+------+-----------------------------------------------------------------------------------------------------------------------------------------+
| id | payload |
+------+-----------------------------------------------------------------------------------------------------------------------------------------+
| 2 | {"active":false,"age":30,"extra":{"levels":[1,2,3]},"name":"bob","profile":{"city":"shanghai"},"tags":["doris"]} |
| 1 | {"active":true,"age":18,"missing":null,"name":"alice","profile":{"city":"beijing","zip":100000},"score":98.5,"tags":["flink","paimon"]} |
| 3 | [1,"mixed",false,null,{"k":"v"}] |
+------+-----------------------------------------------------------------------------------------------------------------------------------------+
3 rows in set (0.13 sec)

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-07-31T09:48:49Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zhangstar333 · 2026-07-31T09:52:09Z

/review

github-actions

Request changes.

Correctness and configuration:

The default Paimon path still sends raw-convertible Variant files to the native reader, which cannot materialize Paimon's physical Variant group into the new compute-v2 destination.
Disabling FileScannerV2 selects the separate legacy JNI implementation, where Variant is unsupported and a SQL-NULL row can write its null marker through address zero.

Compatibility:

In a BE-first rolling upgrade, an old FE can still describe Paimon Variant with the legacy physical type, while the new generic bridge dispatches solely on TYPE_VARIANT and unconditionally casts to ColumnVariantV2.

ABI and lifecycle:

For a confirmed Variant V2 type, the Java buffer capacities, offset domains, metadata IDs, null placeholder, reset/reuse, nested-column handling, synchronous C++ copy, and release/error cleanup are internally consistent; no additional ownership defect was found.

Tests and coverage:

The added unit and regression coverage exercises only the forced-JNI V2 route. It does not cover the default native selection, enable_file_scanner_v2=false, or a legacy physical Variant descriptor.

Performance and security:

No independent performance, authorization, or data-exposure issue was found. The address-zero SQL-NULL write is a memory-safety concern covered inline.

Validation and focus:

Static review only; no builds or tests were run. The complete PR was reviewed, and no additional user-provided focus was supplied. The repository review skill and required AGENTS.md files were not present in this worktree.

zhangstar333 · 2026-08-03T13:05:40Z

run buildall

hello-stephen · 2026-08-03T15:27:50Z

FE UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-08-03T17:36:01Z

BE Regression && UT Coverage Report

Increment line coverage 100.00% (24/24) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.59% (30207/41048)
Line Coverage	57.48% (330643/575227)
Region Coverage	54.42% (275557/506379)
Branch Coverage	55.35% (122735/221734)

hello-stephen · 2026-08-03T17:48:09Z

FE Regression Coverage Report

Increment line coverage 0.00% (0/1) 🎉
Increment coverage report
Complete coverage report

zhangstar333 · 2026-08-04T04:00:31Z

run buildall

hello-stephen · 2026-08-04T05:23:10Z

FE UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-08-04T05:24:50Z

BE UT Coverage Report

Increment line coverage 100.00% (24/24) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	58.31% (24585/42165)
Line Coverage	42.45% (246751/581332)
Region Coverage	38.56% (195023/505817)
Branch Coverage	39.86% (89093/223492)

hello-stephen · 2026-08-04T07:00:17Z

BE Regression && UT Coverage Report

Increment line coverage 100.00% (24/24) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.62% (30316/41179)
Line Coverage	57.50% (332414/578127)
Region Coverage	54.44% (276959/508718)
Branch Coverage	55.35% (123764/223595)

hello-stephen · 2026-08-04T07:12:43Z

FE Regression Coverage Report

Increment line coverage 0.00% (0/1) 🎉
Increment coverage report
Complete coverage report

zhangstar333 · 2026-08-04T08:31:58Z

run buildall

hello-stephen · 2026-08-04T11:54:28Z

BE Regression && UT Coverage Report

Increment line coverage 100.00% (24/24) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.57% (30295/41179)
Line Coverage	57.48% (332290/578127)
Region Coverage	54.32% (276347/508718)
Branch Coverage	55.35% (123750/223595)

hello-stephen · 2026-08-04T12:08:26Z

FE Regression Coverage Report

Increment line coverage 0.00% (0/80) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-08-04T12:47:04Z

FE UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

np

### What problem does this PR solve? Issue Number: closeapache#65086 Related PR: apache#66334, apache#66547, apache#66626 Problem Summary: Forward-port Variant V2 decoding across the Paimon Java scanner and native JNI bridge. The reader propagates nested access paths, materializes encoded Variant bytes into ColumnVariantV2, and accepts Paimon required-value leaves for fallback-only Variant arrays. ### Release note Improve Paimon Variant reads with nested projection and native Variant V2 materialization. ### Check List (For Author) - Test: Unit Test - Added Java vector/projection and BE Parquet schema/reader coverage; BE format/check, header hygiene, and FE checkstyle passed. Final compilation and test execution are deferred until all forward-port picks are complete. - Behavior changed: Yes, Paimon Variant reads support nested projection and fallback-array schemas. - Does this need documentation: No

zhangstar333 requested a review from yiguolei as a code owner July 31, 2026 09:48

github-actionsBot previously requested changes Jul 31, 2026
View reviewed changes

Comment threadbe/src/format/jni/jni_data_bridge.cpp
Comment threadfe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonUtil.java
Comment threadbe/src/format/jni/jni_data_bridge.cpp

zhangstar333 added 3 commits August 3, 2026 20:51

update
dc92746

update
722a9fd

format
9f5736d

zhangstar333 force-pushed the branch-4.1-paimon-variant-jni branch from 79d290d to 9f5736dCompare August 3, 2026 13:02

update case not support
a308221

Gabriel39 added the dev/5.0.x label Aug 4, 2026

update case
3982e06

yiguolei approved these changes Aug 4, 2026
View reviewed changes

yiguolei merged commit 8df1953 into apache:branch-4.1Aug 4, 2026
29 of 32 checks passed

suxiaogang223 mentioned this pull request Sep 1, 2026
[feature](paimon) Forward-port Paimon table writes to master #67395
Open
12 tasks

Uh oh!

Conversation

zhangstar333 commented Jul 31, 2026

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jul 31, 2026

Uh oh!

zhangstar333 commented Jul 31, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zhangstar333 commented Aug 3, 2026

Uh oh!

hello-stephen commented Aug 3, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Aug 3, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Aug 3, 2026

FE Regression Coverage Report

Uh oh!

zhangstar333 commented Aug 4, 2026

Uh oh!

hello-stephen commented Aug 4, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Aug 4, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Aug 4, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Aug 4, 2026

FE Regression Coverage Report

Uh oh!

zhangstar333 commented Aug 4, 2026

Uh oh!

hello-stephen commented Aug 4, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Aug 4, 2026

FE Regression Coverage Report

Uh oh!

hello-stephen commented Aug 4, 2026

FE UT Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants