Uh oh!
There was an error while loading. Please reload this page.
[feature](txn insert) txn insert support insert into select - #31666
Merged
Conversation
doris-robot
commented
Mar 1, 2024
Thank you for your contribution to Apache Doris. |
mymeiyi
commented
Mar 1, 2024
ContributorAuthor
run buildall |
1 similar comment
mymeiyi
commented
Mar 1, 2024
ContributorAuthor
run buildall |
doris-robot
commented
Mar 1, 2024
TPC-H: Total hot run time: 32116 ms |
doris-robot
commented
Mar 1, 2024
TPC-DS: Total hot run time: 187669 ms |
doris-robot
commented
Mar 1, 2024
ClickBench: Total hot run time: 27.03 s |
mymeiyi
commented
Mar 4, 2024
ContributorAuthor
run buildall |
| } else { | ||
| if (ConnectContext.get() != null && ConnectContext.get().isTxnModel()) { | ||
| if (ConnectContext.get() != null && ConnectContext.get().isTxnModel() | ||
| && sink.child() instanceof LogicalInlineTable) { |
Contributor
There was a problem hiding this comment.
In legacy planner, we support insert into t select 1. So if we only support LogicalInlineTable here means we have diff with legacy planner
ContributorAuthor
There was a problem hiding this comment.
The nereids does not support insert into t select 1? It fallback to legacy
| } | ||
| OlapTable olapTable = (OlapTable) targetTableIf; | ||
| String label = this.labelName.orElse( | ||
| ctx.isTxnModel() ? null : String.format("label_%x_%x", ctx.queryId().hi, ctx.queryId().lo)); |
Contributor
There was a problem hiding this comment.
could u add some comment to explain why lable should be null when we in txn model
| TransactionEntry txnEntry = ctx.getTxnEntry(); | ||
| if (txnEntry.isTransactionBegan()) { | ||
| throw new AnalysisException( | ||
| "Transaction insert can not insert into values and insert into select at the same time"); |
Contributor
There was a problem hiding this comment.
why isTransactionBegan means use insert into values with insert into select at same time?
ContributorAuthor
There was a problem hiding this comment.
isTransactionBegan means the insert into select already began a txn.
This function will begin a txn for insert into values.
Currently we forbid it, maybe support it later.
Contributor
There was a problem hiding this comment.
add a TODO coment here, we'll support mix usage of insert into values and insert into select in future.
| boolean isGroupCommit = (parsedStmt != null | ||
| && parsedStmt instanceof LogicalPlanAdapter | ||
| && ((LogicalPlanAdapter) parsedStmt).getLogicalPlan() instanceof InsertIntoTableCommand) | ||
| && !context.isTxnModel(); |
Contributor
There was a problem hiding this comment.
so all insert into without txn model is group commit?
ContributorAuthor
There was a problem hiding this comment.
The logic should be
boolean isGroupCommit = (Config.wait_internal_group_commit_finish
|| context.sessionVariable.isEnableInsertGroupCommit()) && parsedStmt != null
&& parsedStmt instanceof LogicalPlanAdapter
&& ((LogicalPlanAdapter) parsedStmt).getLogicalPlan() instanceof InsertIntoTableCommand;
but some nereids case can not pass.
I add !context.isTxnModel() here becase the regression cases expect txn insert does not force fallback
Contributor
There was a problem hiding this comment.
so the variable name now should not be isGroupCommit? please use a better variable name, and add comments to explain it here
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
mymeiyi
commented
Mar 5, 2024
ContributorAuthor
run buildall |
doris-robot
commented
Mar 5, 2024
TPC-H: Total hot run time: 37666 ms |
doris-robot
commented
Mar 5, 2024
TPC-DS: Total hot run time: 178038 ms |
doris-robot
commented
Mar 5, 2024
ClickBench: Total hot run time: 30.64 s |
doris-robot
commented
Mar 5, 2024
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' |
Contributor
PR approved by at least one committer and no changes requested. |
Contributor
PR approved by anyone and no changes requested. |
mymeiyi
commented
Mar 8, 2024
ContributorAuthor
run buildall |
mymeiyi
commented
Mar 8, 2024
ContributorAuthor
run p0 |
morningman pushed a commit
that referenced
this pull request
Apr 12, 2024
dataroaring pushed a commit
that referenced
this pull request
Jun 5, 2024
## Proposed changes ### Purpose The user doc: https://doris.apache.org/zh-CN/docs/dev/data-operate/import/transaction-load-manual We have supported insert into select(#31666), update(#33034) and delete(#33100) in transaction load. #32980 implements one txn write to one partition more than one rowsets. This pr implements to cloud mode of #32980 ### Implementation #### sub_txn_id see #32980 #### Meta service supports commit txn This process is generally the same as commit_txn, the difference is that he partitions version will plus 1 in multi sub txns. One example: Suppose the table, partition, tablet and version info is: ``` -------------------------------------------- | table | partition | tablet | version | -------------------------------------------- | t1 | t1_p1 | t1_p1.1 | 1 | | t1 | t1_p1 | t1_p1.2 | 1 | | t1 | t1_p2 | t1_p2.1 | 2 | | t2 | t2_p3 | t2_p3.1 | 3 | | t2 | t2_p4 | t2_p4.1 | 4 | -------------------------------------------- ``` Now we commit a txn with 3 sub txns and the tablets are: * sub_txn1: t1_p1.1, t1_p1.2, t1_p2.1 * sub_txn2: t2_p3.1 * sub_txn3: t1_p1.1, t1_p1.2 When commit, the partitions version will be: * sub_txn1: t1_p1(1 -> 2), t1_p2(2 -> 3) * sub_txn2: t2_p3(3 -> 4) * sub_txn3: t1_p1(2 -> 3) After commit, the partitions version will be: * t1: t1_p1(3), t1_p2(3) * t2: t2_p3(4), t2_p4(4) #### Meta service support generate sub_txn_id by `begin_sub_txn`
dataroaring pushed a commit
that referenced
this pull request
Jun 7, 2024
## Proposed changes ### Purpose The user doc: https://doris.apache.org/zh-CN/docs/dev/data-operate/import/transaction-load-manual We have supported insert into select(#31666), update(#33034) and delete(#33100) in transaction load. #32980 implements one txn write to one partition more than one rowsets. This pr implements to cloud mode of #32980 ### Implementation #### sub_txn_id see #32980 #### Meta service supports commit txn This process is generally the same as commit_txn, the difference is that he partitions version will plus 1 in multi sub txns. One example: Suppose the table, partition, tablet and version info is: ``` -------------------------------------------- | table | partition | tablet | version | -------------------------------------------- | t1 | t1_p1 | t1_p1.1 | 1 | | t1 | t1_p1 | t1_p1.2 | 1 | | t1 | t1_p2 | t1_p2.1 | 2 | | t2 | t2_p3 | t2_p3.1 | 3 | | t2 | t2_p4 | t2_p4.1 | 4 | -------------------------------------------- ``` Now we commit a txn with 3 sub txns and the tablets are: * sub_txn1: t1_p1.1, t1_p1.2, t1_p2.1 * sub_txn2: t2_p3.1 * sub_txn3: t1_p1.1, t1_p1.2 When commit, the partitions version will be: * sub_txn1: t1_p1(1 -> 2), t1_p2(2 -> 3) * sub_txn2: t2_p3(3 -> 4) * sub_txn3: t1_p1(2 -> 3) After commit, the partitions version will be: * t1: t1_p1(3), t1_p2(3) * t2: t2_p3(4), t2_p4(4) #### Meta service support generate sub_txn_id by `begin_sub_txn`
HappenLee pushed a commit
to HappenLee/incubator-doris
that referenced
this pull request
Apr 24, 2026
HappenLee pushed a commit
to HappenLee/incubator-doris
that referenced
this pull request
Apr 24, 2026
## Proposed changes ### Purpose The user doc: https://doris.apache.org/zh-CN/docs/dev/data-operate/import/transaction-load-manual We have supported insert into select(apache#31666), update(apache#33034) and delete(apache#33100) in transaction load. apache#32980 implements one txn write to one partition more than one rowsets. This pr implements to cloud mode of apache#32980 ### Implementation #### sub_txn_id see apache#32980 #### Meta service supports commit txn This process is generally the same as commit_txn, the difference is that he partitions version will plus 1 in multi sub txns. One example: Suppose the table, partition, tablet and version info is: ``` -------------------------------------------- | table | partition | tablet | version | -------------------------------------------- | t1 | t1_p1 | t1_p1.1 | 1 | | t1 | t1_p1 | t1_p1.2 | 1 | | t1 | t1_p2 | t1_p2.1 | 2 | | t2 | t2_p3 | t2_p3.1 | 3 | | t2 | t2_p4 | t2_p4.1 | 4 | -------------------------------------------- ``` Now we commit a txn with 3 sub txns and the tablets are: * sub_txn1: t1_p1.1, t1_p1.2, t1_p2.1 * sub_txn2: t2_p3.1 * sub_txn3: t1_p1.1, t1_p1.2 When commit, the partitions version will be: * sub_txn1: t1_p1(1 -> 2), t1_p2(2 -> 3) * sub_txn2: t2_p3(3 -> 4) * sub_txn3: t1_p1(2 -> 3) After commit, the partitions version will be: * t1: t1_p1(3), t1_p2(3) * t2: t2_p3(4), t2_p4(4) #### Meta service support generate sub_txn_id by `begin_sub_txn`
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
Transaction insert support:
Limit:
insert into selectandinsert into valuescan be used at one transaction.Will support in later pr.
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...