Skip to content

[feature](mtmv)Support iceberg partition refresh. - #44726

Merged
Jibing-Li merged 1 commit into
apache:masterfrom
Jibing-Li:iceberg
Dec 16, 2024
Merged

[feature](mtmv)Support iceberg partition refresh.#44726
Jibing-Li merged 1 commit into
apache:masterfrom
Jibing-Li:iceberg

Conversation

@Jibing-Li

@Jibing-LiJibing-Li commented Nov 28, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Previously, when using Iceberg to create MTMV, it was not possible to perceive changes in partition lists and data, so only refresh materialized view mv1 complete could be used to force full refresh.

This PR obtains the partition list of Iceberg, the last update time of the partition, and the latest snapshotId of the table.

Therefore, MTMV can be partition based on Iceberg tables and perceive changes in data, automatically refreshing partitions

For now, we only support single partition column table and the partition transform must one of hour, day, month or year.
Will support Identity transform soon.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot

doris-robot commented Nov 28, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Jibing-Li
Jibing-Liforce-pushed the iceberg branch 3 times, most recently from 870c5ab to ea79a6aCompareNovember 28, 2024 09:16
@Jibing-Li
Jibing-Li marked this pull request as ready for review November 28, 2024 09:17
@Jibing-Li

Jibing-Li commented Nov 28, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

morrySnow
morrySnow reviewed Nov 29, 2024
View reviewed changes
Comment threadfe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java Outdated
zddr
zddr reviewed Nov 29, 2024
View reviewed changes
Comment threadfe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java
Comment threadregression-test/suites/mtmv_p0/test_iceberg_mtmv.groovy Outdated
Comment threadregression-test/suites/mtmv_p0/test_iceberg_mtmv.groovy
@Jibing-Li
Jibing-Liforce-pushed the iceberg branch 10 times, most recently from 235a77f to 7ef6b3aCompareDecember 6, 2024 06:08
@Jibing-Li

Jibing-Li commented Dec 6, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

zddr
zddr reviewed Dec 10, 2024
View reviewed changes
Comment threadfe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelatedTableIf.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java
Comment threadfe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergPartitionInfo.java
Comment threadfe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java
@Jibing-Li
Jibing-Liforce-pushed the iceberg branch 4 times, most recently from 708edf0 to 1df4979CompareDecember 10, 2024 08:28
@Jibing-Li

Jibing-Li commented Dec 10, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

zddr
zddr approved these changes Dec 10, 2024
View reviewed changes

@zddrzddr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

github-actionsBot commented Dec 10, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@Jibing-Li

Jibing-Li commented Dec 10, 2024

Copy link
Copy Markdown
ContributorAuthor

run p0

morrySnow
morrySnow reviewed Dec 13, 2024
View reviewed changes
Comment threadfe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java Outdated
@Jibing-Li

Jibing-Li commented Dec 16, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actions

github-actionsBot commented Dec 16, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Dec 16, 2024
@Jibing-Li
Jibing-Li merged commit f49e609 into apache:masterDec 16, 2024
github-actionsBot pushed a commit that referenced this pull request Dec 16, 2024
@Jibing-LiYour Name
[feature](mtmv)Support iceberg partition refresh. (#44726)
564b3b6
### What problem does this PR solve?
Previously, when using Iceberg to create MTMV, it was not possible to
perceive changes in partition lists and data, so only ```refresh
materialized view mv1 complete ```could be used to force full refresh.
This PR obtains the partition list of Iceberg, the last update time of
the partition, and the latest snapshotId of the table.
Therefore, MTMV can be partition based on Iceberg tables and perceive
changes in data, automatically refreshing partitions
For now, we only support single partition column table and the partition
transform must one of hour, day, month or year.
Will support Identity transform soon.
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
@github-actionsgithub-actionsBot mentioned this pull request Dec 16, 2024
Closed
github-actionsBot pushed a commit that referenced this pull request Dec 16, 2024
@Jibing-LiYour Name
[feature](mtmv)Support iceberg partition refresh. (#44726)
7ec73c1
### What problem does this PR solve?
Previously, when using Iceberg to create MTMV, it was not possible to
perceive changes in partition lists and data, so only ```refresh
materialized view mv1 complete ```could be used to force full refresh.
This PR obtains the partition list of Iceberg, the last update time of
the partition, and the latest snapshotId of the table.
Therefore, MTMV can be partition based on Iceberg tables and perceive
changes in data, automatically refreshing partitions
For now, we only support single partition column table and the partition
transform must one of hour, day, month or year.
Will support Identity transform soon.
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
@github-actionsgithub-actionsBot mentioned this pull request Dec 16, 2024
Closed
@Jibing-Li
Jibing-Li deleted the iceberg branch December 16, 2024 08:31
zddr pushed a commit to zddr/incubator-doris that referenced this pull request Jun 19, 2025
@Jibing-Li@zddr
[feature](mtmv)Support iceberg partition refresh. (apache#44726)
22d29ef
### What problem does this PR solve?
Previously, when using Iceberg to create MTMV, it was not possible to
perceive changes in partition lists and data, so only ```refresh
materialized view mv1 complete ```could be used to force full refresh.
This PR obtains the partition list of Iceberg, the last update time of
the partition, and the latest snapshotId of the table.
Therefore, MTMV can be partition based on Iceberg tables and perceive
changes in data, automatically refreshing partitions
For now, we only support single partition column table and the partition
transform must one of hour, day, month or year.
Will support Identity transform soon.
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
zddr pushed a commit to zddr/incubator-doris that referenced this pull request Jun 19, 2025
@Jibing-Li@zddr
[feature](mtmv)Support iceberg partition refresh. (apache#44726)
e82fc3b
Previously, when using Iceberg to create MTMV, it was not possible to
perceive changes in partition lists and data, so only ```refresh
materialized view mv1 complete ```could be used to force full refresh.
This PR obtains the partition list of Iceberg, the last update time of
the partition, and the latest snapshotId of the table.
Therefore, MTMV can be partition based on Iceberg tables and perceive
changes in data, automatically refreshing partitions
For now, we only support single partition column table and the partition
transform must one of hour, day, month or year.
Will support Identity transform soon.
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
None
@zddrzddr mentioned this pull request Jun 20, 2025
Merged
morrySnow pushed a commit that referenced this pull request Jun 23, 2025
@zddr@Jibing-Li@BePPPower
branch-3.1: [enhance](mtmv)MTMV support iceberg/hudi/paimon (#52040)
4234a27
Cherry-pick from
#43959#44419#44415#44567#44673#44998#45273#44911#44726#45652#45659#46257#46641#47026#47166#48172#49956#50979
---------
Co-authored-by: James <lijibing@selectdb.com>
Co-authored-by: Tiewei Fang <fangtiewei@selectdb.com>
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@Jibing-Li@HappenLee
[feature](mtmv)Support iceberg partition refresh. (apache#44726)
af53a73
### What problem does this PR solve?
Previously, when using Iceberg to create MTMV, it was not possible to
perceive changes in partition lists and data, so only ```refresh
materialized view mv1 complete ```could be used to force full refresh.
This PR obtains the partition list of Iceberg, the last update time of
the partition, and the latest snapshotId of the table.
Therefore, MTMV can be partition based on Iceberg tables and perceive
changes in data, automatically refreshing partitions
For now, we only support single partition column table and the partition
transform must one of hour, day, month or year.
Will support Identity transform soon.
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zddrzddrzddr approved these changes
@morrySnowmorrySnowmorrySnow approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/3.1.0-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Jibing-Li@doris-robot@zddr@morrySnow