Uh oh!
There was an error while loading. Please reload this page.
[Enhancement](auto partition) Auto partition could use auto bucket - #56921
Conversation
hello-stephen
commented
Oct 13, 2025
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
zclllyybb
commented
Oct 13, 2025
run buildall |
There was a problem hiding this comment.
Pull Request Overview
This PR enhances auto partition functionality to support auto bucket calculation for auto range partitions while forbidding the combination of auto list partition with auto bucket. The changes enable better bucket sizing for new partitions based on historical partition data patterns.
Key changes:
- Added validation to prevent using auto bucket with auto list partition
- Implemented auto bucket calculation for auto range partitions using historical data analysis
- Added comprehensive test coverage for the new functionality
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| test_auto_partition_behavior.groovy | Adds regression test to verify auto list partition + auto bucket combination is forbidden |
| PartitionExprUtilTest.java | New unit tests for auto bucket calculation with historical partition data |
| CreateTableInfo.java | Validation logic to prevent auto bucket with auto list partition |
| DynamicPartitionScheduler.java | Made helper methods public for bucket calculation |
| PartitionExprUtil.java | Core logic for auto bucket calculation in auto range partitions |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
zclllyybb
commented
Oct 13, 2025
run buildall |
zclllyybb
commented
Oct 13, 2025
run buildall |
doris-robot
commented
Oct 13, 2025
TPC-DS: Total hot run time: 190630 ms |
doris-robot
commented
Oct 13, 2025
ClickBench: Total hot run time: 31.17 s |
hello-stephen
commented
Oct 13, 2025
FE Regression Coverage ReportIncrement line coverage |
PR approved by at least one committer and no changes requested. |
PR approved by anyone and no changes requested. |
zclllyybb
commented
Oct 14, 2025
run buildall |
doris-robot
commented
Oct 14, 2025
TPC-DS: Total hot run time: 189592 ms |
doris-robot
commented
Oct 14, 2025
ClickBench: Total hot run time: 30.31 s |
hello-stephen
commented
Oct 14, 2025
FE UT Coverage ReportIncrement line coverage |
hello-stephen
commented
Oct 14, 2025
FE Regression Coverage ReportIncrement line coverage |
…2970) doc of apache/doris#56921 ## Versions - [x] dev - [x] 3.0 - [ ] 2.1 - [ ] 2.0 ## Languages - [x] Chinese - [x] English ## Docs Checklist - [ ] Checked by AI - [ ] Test Cases Built
Uh oh!
There was an error while loading. Please reload this page.
zclllyybb
commented
Oct 15, 2025
run buildall |
zclllyybb
commented
Oct 15, 2025
run buildall |
doris-robot
commented
Oct 15, 2025
TPC-DS: Total hot run time: 190478 ms |
doris-robot
commented
Oct 15, 2025
ClickBench: Total hot run time: 30.54 s |
hello-stephen
commented
Oct 15, 2025
FE UT Coverage ReportIncrement line coverage |
PR approved by at least one committer and no changes requested. |
Uh oh!
There was an error while loading. Please reload this page.
…56921) 1. forbid auto list partition + auto bucket 2. support auto range partition + auto bucket to calculate proper bucket for new partition. base on the assumption of incremental load of table.
…pache#56921) 1. forbid auto list partition + auto bucket 2. support auto range partition + auto bucket to calculate proper bucket for new partition. base on the assumption of incremental load of table.
Related PR: introduced by #56921 Problem Summary: fix concurrency bug when dynamic partition visit partition items at the same time with modifying partitions: ```java 2026-01-12 15:31:46,108 ERROR (thrift-server-pool-31|451) [ProcessFunction.process():47] Internal error processing createPartition java.lang.reflect.UndeclaredThrowableException at jdk.proxy2/jdk.proxy2.$Proxy27.createPartition(Unknown Source) at org.apache.doris.thrift.FrontendService$Processor$createPartition.getResult(FrontendService.java:4993) at org.apache.doris.thrift.FrontendService$Processor$createPartition.getResult(FrontendService.java:4973) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: java.lang.reflect.InvocationTargetException at jdk.internal.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.apache.doris.service.FeServer.lambda$start$0(FeServer.java:60) ... 9 more Caused by: java.util.ConcurrentModificationException at java.base/java.util.HashMap$HashIterator.nextNode(HashMap.java:1597) at java.base/java.util.HashMap$EntryIterator.next(HashMap.java:1630) at java.base/java.util.HashMap$EntryIterator.next(HashMap.java:1628) at java.base/java.util.AbstractCollection.toArray(AbstractCollection.java:146) at java.base/java.util.ArrayList.<init>(ArrayList.java:181) at org.apache.doris.clone.DynamicPartitionScheduler.getHistoricalPartitions(DynamicPartitionScheduler.java:209) at org.apache.doris.common.util.AutoBucketCalculator.calculateAutoBuckets(AutoBucketCalculator.java:142) at org.apache.doris.common.util.AutoBucketCalculator.calculateAutoBucketsWithBoundsCheck(AutoBucketCalculator.java:190) at org.apache.doris.analysis.PartitionExprUtil.getAddPartitionClauseFromPartitionValues(PartitionExprUtil.java:202) at org.apache.doris.service.FrontendServiceImpl.createPartition(FrontendServiceImpl.java:3622) ... 13 more ``` all these visit operations should under table's lock.
Related PR: introduced by #56921 Problem Summary: fix concurrency bug when dynamic partition visit partition items at the same time with modifying partitions: ```java 2026-01-12 15:31:46,108 ERROR (thrift-server-pool-31|451) [ProcessFunction.process():47] Internal error processing createPartition java.lang.reflect.UndeclaredThrowableException at jdk.proxy2/jdk.proxy2.$Proxy27.createPartition(Unknown Source) at org.apache.doris.thrift.FrontendService$Processor$createPartition.getResult(FrontendService.java:4993) at org.apache.doris.thrift.FrontendService$Processor$createPartition.getResult(FrontendService.java:4973) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: java.lang.reflect.InvocationTargetException at jdk.internal.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.apache.doris.service.FeServer.lambda$start$0(FeServer.java:60) ... 9 more Caused by: java.util.ConcurrentModificationException at java.base/java.util.HashMap$HashIterator.nextNode(HashMap.java:1597) at java.base/java.util.HashMap$EntryIterator.next(HashMap.java:1630) at java.base/java.util.HashMap$EntryIterator.next(HashMap.java:1628) at java.base/java.util.AbstractCollection.toArray(AbstractCollection.java:146) at java.base/java.util.ArrayList.<init>(ArrayList.java:181) at org.apache.doris.clone.DynamicPartitionScheduler.getHistoricalPartitions(DynamicPartitionScheduler.java:209) at org.apache.doris.common.util.AutoBucketCalculator.calculateAutoBuckets(AutoBucketCalculator.java:142) at org.apache.doris.common.util.AutoBucketCalculator.calculateAutoBucketsWithBoundsCheck(AutoBucketCalculator.java:190) at org.apache.doris.analysis.PartitionExprUtil.getAddPartitionClauseFromPartitionValues(PartitionExprUtil.java:202) at org.apache.doris.service.FrontendServiceImpl.createPartition(FrontendServiceImpl.java:3622) ... 13 more ``` all these visit operations should under table's lock.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)