branch-4.1: [feature](iceberg) Implements iceberg update delete merge into functionality. #60482 by kaka11chen · Pull Request #61818 · apache/doris

kaka11chen · 2026-03-27T10:00:20Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

Cherry pick #60482

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-03-27T10:00:27Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

kaka11chen · 2026-03-27T10:00:28Z

run buildall

doris-robot · 2026-03-27T10:21:03Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	78.44% (1786/2277)
Line Coverage	64.19% (32004/49856)
Region Coverage	65.06% (16016/24619)
Branch Coverage	55.58% (8528/15344)

hello-stephen · 2026-03-27T11:09:45Z

FE UT Coverage Report

Increment line coverage 42.40% (803/1894) 🎉
Increment coverage report
Complete coverage report

…onality. (apache#60482) [feature] (iceberg) Implements iceberg update & delete & merge into. 1. **Nereids Planner**: - Introduced parser and planner support for `DELETE`, `UPDATE`, and `MERGE INTO` targeting Iceberg tables. - Added specific plan nodes and rules (`IcebergDeleteCommand`, `IcebergMergeCommand`, `IcebergUpdateCommand`, `Logical/PhysicalIcebergDeleteSink`, `Logical/PhysicalIcebergMergeSink`). 2. **Frontend & Transaction**: - Handled Iceberg DML transactional context and snapshot commits via `IcebergTransaction`. - Implemented conflict detection and resolution utilities for concurrent modifications (`IcebergConflictDetectionFilterUtils`). - Added support for Iceberg metadata columns (e.g., `_file_path`, `_pos`) via `IcebergMetadataColumn` to accurately locate deleting records. 3. **Backend Execution Sinks**: - Introduced `VIcebergDeleteSink` and `VIcebergMergeSink` sink operators. - Added `VIcebergDeleteFileWriter` mechanism to support writing Position Delete files natively inside Doris BE. - Implemented routing and data shuffling logic via `MergePartitioner` and `IcebergPartitionFunction` to properly align input rows with Iceberg's partitioning constraints during writes. 4. **Scanner & Execution Decoupling**: - Adjusted `FileScanner` and `GenericReader` to natively support row-level operations (like generating `$row_id`), feeding precise coordinates for DML modifications. 5. **Quality Assurance**: - Added comprehensive logic/physical plan tests (e.g. `IcebergDDLAndDMLPlanTest`). - Extensive coverage of the sink components in BE Unit Tests (`viceberg_merge_sink_test.cpp`, `viceberg_delete_sink_test.cpp`, etc.). - Introduced standard regression test suites covering basic/advanced usages and edge cases for merge/update/delete (`regression-test/suites/external_table_p0/iceberg/dml/*`).

kaka11chen · 2026-03-27T14:49:13Z

run buildall

doris-robot · 2026-03-27T15:35:29Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	78.44% (1786/2277)
Line Coverage	64.24% (32026/49856)
Region Coverage	65.09% (16025/24619)
Branch Coverage	55.57% (8526/15344)

hello-stephen · 2026-03-27T16:14:20Z

FE UT Coverage Report

Increment line coverage 42.40% (803/1894) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2026-03-27T17:09:10Z

BE UT Coverage Report

Increment line coverage 52.94% (954/1802) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.94% (19643/37103)
Line Coverage	36.29% (183801/506479)
Region Coverage	32.61% (142166/436004)
Branch Coverage	33.80% (62481/184857)

hello-stephen · 2026-03-27T17:24:28Z

BE Regression && UT Coverage Report

Increment line coverage 80.56% (1450/1800) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	72.90% (26476/36319)
Line Coverage	56.24% (283882/504784)
Region Coverage	53.79% (236670/439974)
Branch Coverage	55.49% (102872/185385)

@0x0

…ead (#66506) ### What problem does this PR solve? Problem Summary: coredump: ``` *** Query id: 8bxxx-xxx *** *** is nereids: 1 *** *** tablet id: 0 *** *** Aborted at 1785897717 (unix time) try "date -d @1785897717" if you are using GNU date *** *** Current BE git commitID: 2607291 *** *** SIGSEGV address not mapped to object (@0x0) received by PID 1002 (TID 3954234 OR 0x7f42d110e6c0) from PID 0; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) in /usr/local/service/doris/lib/be/doris_be 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.10] in /usr/local/jdk/lib/server/libjvm.so 2# JVM_handle_linux_signal in /usr/local/jdk/lib/server/libjvm.so 3# 0x00007F8C17A4ABB0 in /lib64/libc.so.6 4# doris::Block::filter_block_internal(doris::Block*, std::vector<unsigned int, std::allocator<unsigned int> > const&, doris::PODArray<unsigned char, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, true>, 16ul, 15ul> const&) in /usr/local/service/doris/lib/be/doris_be 5# doris::Block::filter_block_internal(doris::Block*, doris::PODArray<unsigned char, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, true>, 16ul, 15ul> const&, unsigned int) in /usr/local/service/doris/lib/be/doris_be 6# doris::IcebergTableReader::get_next_block_inner(doris::Block*, unsigned long*, bool*) in /usr/local/service/doris/lib/be/doris_be 7# doris::TableFormatReader::get_next_block(doris::Block*, unsigned long*, bool*) in /usr/local/service/doris/lib/be/doris_be 8# doris::FileScanner::_get_block_wrapped(doris::RuntimeState*, doris::Block*, bool*) in /usr/local/service/doris/lib/be/doris_be 9# doris::FileScanner::_get_block_impl(doris::RuntimeState*, doris::Block*, bool*) in /usr/local/service/doris/lib/be/doris_be 10# doris::Scanner::get_block(doris::RuntimeState*, doris::Block*, bool*) in /usr/local/service/doris/lib/be/doris_be 11# doris::Scanner::get_block_after_projects(doris::RuntimeState*, doris::Block*, bool*) in /usr/local/service/doris/lib/be/doris_be 12# doris::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>) in /usr/local/service/doris/lib/be/doris_be 13# std::_Function_handler<bool (), doris::ScannerScheduler::submit(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>)::$_0::operator()() const::{lambda()#1}>::_M_invoke(std::_Any_data const&) in /usr/local/service/doris/lib/be/doris_be 14# doris::ScannerSplitRunner::process_for(std::chrono::duration<long, std::ratio<1l, 1000000000l> >) in /usr/local/service/doris/lib/be/doris_be 15# doris::PrioritizedSplitRunner::process() in /usr/local/service/doris/lib/be/doris_be 16# doris::TimeSharingTaskExecutor::_dispatch_thread() in /usr/local/service/doris/lib/be/doris_be 17# doris::Thread::supervise_thread(void*) in /usr/local/service/doris/lib/be/doris_be 18# start_thread in /lib64/libc.so.6 19# __GI___clone3 in /lib64/libc.so.6 ``` The master branch is fine. This bug only exists in 4.1 branch and introduce by: #61818 This commit only pick this line from the pr in master: #60482 ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [x] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [x] No. - [ ] Yes.  - Does this need documentation? - [x] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

kaka11chen requested a review from yiguolei as a code owner March 27, 2026 10:00

kaka11chen force-pushed the cherry-pick-60482_4.1 branch from 010249a to 142041bCompare March 27, 2026 14:48

morningman changed the title ~~[feature](iceberg) Implements iceberg update delete merge into functi…~~branch-4.1: [feature](iceberg) Implements iceberg update delete merge into functi…Mar 27, 2026

morningman changed the title ~~branch-4.1: [feature](iceberg) Implements iceberg update delete merge into functi…~~branch-4.1: [feature](iceberg) Implements iceberg update delete merge into functionality. #60482Mar 27, 2026

yiguolei approved these changes Mar 28, 2026
View reviewed changes

yiguolei merged commit 463b549 into apache:branch-4.1Mar 28, 2026
27 of 30 checks passed

xy720 mentioned this pull request Aug 5, 2026
[Bug](iceberg) fix BE core when reading MOR table with rowid + lazy read #66506
Merged
16 tasks

Navigation Menu

Uh oh!

branch-4.1: [feature](iceberg) Implements iceberg update delete merge into functionality. #60482 - #61818

branch-4.1: [feature](iceberg) Implements iceberg update delete merge into functionality. #60482#61818
yiguolei merged 1 commit into
apache:branch-4.1from
kaka11chen:cherry-pick-60482_4.1

kaka11chen commented Mar 27, 2026

Uh oh!

hello-stephen commented Mar 27, 2026

Uh oh!

kaka11chen commented Mar 27, 2026

Uh oh!

doris-robot commented Mar 27, 2026

Uh oh!

hello-stephen commented Mar 27, 2026

Uh oh!

kaka11chen commented Mar 27, 2026

Uh oh!

doris-robot commented Mar 27, 2026

Uh oh!

hello-stephen commented Mar 27, 2026

Uh oh!

doris-robot commented Mar 27, 2026

Uh oh!

hello-stephen commented Mar 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

kaka11chen commented Mar 27, 2026

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Mar 27, 2026

Uh oh!

kaka11chen commented Mar 27, 2026

Uh oh!

doris-robot commented Mar 27, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Mar 27, 2026

FE UT Coverage Report

Uh oh!

kaka11chen commented Mar 27, 2026

Uh oh!

doris-robot commented Mar 27, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Mar 27, 2026

FE UT Coverage Report

Uh oh!

doris-robot commented Mar 27, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Mar 27, 2026

BE Regression && UT Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants