[Fix](partial update) Persist partial_update_info in RocksDB in case of BE restart after a partial update has commited by bobhan1 · Pull Request #38331 · apache/doris

bobhan1 · 2024-07-24T14:53:20Z

Proposed changes

If a partial update has conflict with another load during publish phase, it should combine the two load's data into one to get the corrrect result. This procedure needs partial update info. But If BE crashed after the partial update load has committed, the partial update info will be missing becasuse it's not persisted and will not be restored in DataDir::load(). This PR persists partial update info in RocksDB before the txn is commited and remove it after the publish phase.
Before #25147, partial update info is persisted with tablet_schema in RocksDB. #25147 split partial update info from tablet schema but forget to handle the persistence logic.

branch-2.1-pick: #39035
branch-2.0-pick: #39078

doris-robot · 2024-07-24T14:53:24Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions

clang-tidy made some suggestions

github-actions · 2024-07-24T15:00:45Z

+
+#include "olap/partial_update_info.h"
+
+#include <gen_cpp/olap_file.pb.h>


warning: 'gen_cpp/olap_file.pb.h' file not found [clang-diagnostic-error]
#include<gen_cpp/olap_file.pb.h> ^

github-actions · 2024-07-24T15:00:45Z

 struct PartialUpdateInfo {
 void init(const TabletSchema& tablet_schema, bool partial_update,
- const std::set<string>& partial_update_cols, bool is_strict_mode,
+ const std::set<std::string>& partial_update_cols, bool is_strict_mode,


warning: no template named 'set' in namespace 'std' [clang-diagnostic-error]
const std::set<std::string>& partial_update_cols, bool is_strict_mode, ^

github-actions · 2024-07-24T15:00:45Z

 #ifndef DORIS_BE_SRC_OLAP_ROWSET_ROWSET_META_MANAGER_H
 #define DORIS_BE_SRC_OLAP_ROWSET_ROWSET_META_MANAGER_H

+#include <gen_cpp/olap_file.pb.h>


warning: 'gen_cpp/olap_file.pb.h' file not found [clang-diagnostic-error]
#include<gen_cpp/olap_file.pb.h> ^

bobhan1 · 2024-07-25T02:35:56Z

run buildall

bobhan1 · 2024-07-25T06:24:25Z

run buildall

bobhan1 · 2024-07-25T11:54:29Z

run buildall

bobhan1 · 2024-07-29T02:42:30Z

run buildall

dataroaring · 2024-08-01T02:18:15Z

run buildall

tmp tmp tmp tmp finish finish case tmp fix fix complie tmp tmp tmp tmp

zhannngchen

LGTM

github-actions · 2024-08-01T06:23:17Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-08-01T06:23:19Z

PR approved by anyone and no changes requested.

bobhan1 · 2024-08-01T06:34:13Z

run buildall

zhannngchen

LGTM

github-actions · 2024-08-02T13:02:05Z

PR approved by at least one committer and no changes requested.

hust-hhb

LGTM

… in RocksDB in case of BE restart after a partial update has commited #38331" (#39035) picks #38331 and #39066

…inter (#39066) fix member call on null pointer in #38331

… in RocksDB in case of BE restart after a partial update has commited #38331" (#39078) picks #38331 and #39066

…of BE restart after a partial update has commited (#38331) ## Proposed changes If a partial update has conflict with another load during publish phase, it should combine the two load's data into one to get the corrrect result. This procedure needs partial update info. But If BE crashed after the partial update load has committed, the partial update info will be missing becasuse it's not persisted and will not be restored in `DataDir::load()`. This PR persists partial update info in RocksDB before the txn is commited and remove it after the publish phase. Before #25147, partial update info is persisted with tablet_schema in RocksDB. #25147 split partial update info from tablet schema but forget to handle the persistence logic.

…inter (#39066) fix member call on null pointer in #38331

…of BE restart after a partial update has commited (apache#38331) ## Proposed changes If a partial update has conflict with another load during publish phase, it should combine the two load's data into one to get the corrrect result. This procedure needs partial update info. But If BE crashed after the partial update load has committed, the partial update info will be missing becasuse it's not persisted and will not be restored in `DataDir::load()`. This PR persists partial update info in RocksDB before the txn is commited and remove it after the publish phase. Before apache#25147, partial update info is persisted with tablet_schema in RocksDB. apache#25147 split partial update info from tablet schema but forget to handle the persistence logic.

…inter (apache#39066) fix member call on null pointer in apache#38331

…of BE restart after a partial update has commited (#38331) ## Proposed changes If a partial update has conflict with another load during publish phase, it should combine the two load's data into one to get the corrrect result. This procedure needs partial update info. But If BE crashed after the partial update load has committed, the partial update info will be missing becasuse it's not persisted and will not be restored in `DataDir::load()`. This PR persists partial update info in RocksDB before the txn is commited and remove it after the publish phase. Before #25147, partial update info is persisted with tablet_schema in RocksDB. #25147 split partial update info from tablet schema but forget to handle the persistence logic.

…inter (#39066) fix member call on null pointer in #38331

… in RocksDB in case of BE restart after a partial update has commited apache#38331" (apache#39078) picks apache#38331 and apache#39066

…of BE restart after a partial update has commited (apache#38331) ## Proposed changes If a partial update has conflict with another load during publish phase, it should combine the two load's data into one to get the corrrect result. This procedure needs partial update info. But If BE crashed after the partial update load has committed, the partial update info will be missing becasuse it's not persisted and will not be restored in `DataDir::load()`. This PR persists partial update info in RocksDB before the txn is commited and remove it after the publish phase. Before apache#25147, partial update info is persisted with tablet_schema in RocksDB. apache#25147 split partial update info from tablet schema but forget to handle the persistence logic.

…inter (apache#39066) fix member call on null pointer in apache#38331

github-actionsBot added the meta-change label Jul 24, 2024

bobhan1 force-pushed the persist-partial-update-info branch from 236a525 to e79f107Compare July 24, 2024 14:57

github-actionsBot reviewed Jul 24, 2024
View reviewed changes

bobhan1 force-pushed the persist-partial-update-info branch from ad2d9a9 to efe2ea6Compare July 25, 2024 11:54

bobhan1 force-pushed the persist-partial-update-info branch from efe2ea6 to 762e3efCompare July 29, 2024 02:15

zhannngchen reviewed Jul 30, 2024
View reviewed changes

Comment threadbe/src/olap/rowset/rowset_meta_manager.cpp Outdated
Comment threadbe/src/olap/rowset/rowset_meta_manager.cpp Outdated
Comment threadbe/src/olap/txn_manager.cpp Outdated
Comment threadbe/src/olap/storage_engine.cpp Outdated

bobhan1 force-pushed the persist-partial-update-info branch 2 times, most recently from 68d404b to 4095139Compare July 31, 2024 03:32

bobhan1 force-pushed the persist-partial-update-info branch from 4095139 to f9af921Compare August 1, 2024 05:54

github-actionsBot added the doing label Aug 1, 2024

update
d4b8f3b
tmp tmp tmp tmp finish finish case tmp fix fix complie tmp tmp tmp tmp

bobhan1 force-pushed the persist-partial-update-info branch from f9af921 to d4b8f3bCompare August 1, 2024 06:08

zhannngchen previously approved these changes Aug 1, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 1, 2024

github-actionsBot added the reviewed label Aug 1, 2024

bobhan1 dismissed zhannngchen’s stale review via be9266fAugust 1, 2024 06:24

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Aug 1, 2024

fix deserilize
628fb44

bobhan1 force-pushed the persist-partial-update-info branch from be9266f to 628fb44Compare August 1, 2024 06:33

zhannngchen previously approved these changes Aug 1, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 1, 2024

Yukang-Lian approved these changes Aug 5, 2024
View reviewed changes

hust-hhb approved these changes Aug 5, 2024
View reviewed changes

dataroaring added dev/2.1.x dev/2.0.x labels Aug 7, 2024

dataroaring merged commit 0b834be into apache:masterAug 7, 2024

zhannngchen pushed a commit that referenced this pull request Aug 8, 2024
[branch-2.1] Picks "[Fix](partial update) Persist partial_update_info…
1fbfb81
… in RocksDB in case of BE restart after a partial update has commited #38331" (#39035) picks #38331 and #39066

zhannngchen pushed a commit that referenced this pull request Aug 8, 2024
[Fix](merge-on-write) Fix UBSan error cause by member call on null po…
7ead170
…inter (#39066) fix member call on null pointer in #38331

dataroaring added dev/2.1.6-merged and removed dev/2.1.x labels Aug 8, 2024

dataroaring pushed a commit that referenced this pull request Aug 9, 2024
[branch-2.0] Picks "[Fix](partial update) Persist partial_update_info…
bf11d48
… in RocksDB in case of BE restart after a partial update has commited #38331" (#39078) picks #38331 and #39066

dataroaring added dev/2.0.15-merged dev/3.0.x and removed dev/2.0.x labels Aug 9, 2024

dataroaring pushed a commit that referenced this pull request Aug 11, 2024
[Fix](merge-on-write) Fix UBSan error cause by member call on null po…
4104b0e
…inter (#39066) fix member call on null pointer in #38331

dataroaring added dev/3.0.2-merged and removed dev/3.0.x labels Aug 12, 2024

wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Aug 14, 2024
[Fix](merge-on-write) Fix UBSan error cause by member call on null po…
0e9e99c
…inter (apache#39066) fix member call on null pointer in apache#38331

dataroaring pushed a commit that referenced this pull request Aug 16, 2024
[Fix](merge-on-write) Fix UBSan error cause by member call on null po…
5ebb564
…inter (#39066) fix member call on null pointer in #38331

gavinchou mentioned this pull request Oct 13, 2024
Release Note 3.0.2 #41558
Closed

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[Fix](merge-on-write) Fix UBSan error cause by member call on null po…
aab1a55
…inter (apache#39066) fix member call on null pointer in apache#38331


		#include "olap/partial_update_info.h"

		#include <gen_cpp/olap_file.pb.h>

Uh oh!

Conversation

bobhan1 commented Jul 24, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented Jul 24, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBotJul 24, 2024

Choose a reason for hiding this comment

Uh oh!

github-actionsBotJul 24, 2024

Choose a reason for hiding this comment

Uh oh!

github-actionsBotJul 24, 2024

Choose a reason for hiding this comment

Uh oh!

bobhan1 commented Jul 25, 2024

Uh oh!

bobhan1 commented Jul 25, 2024

Uh oh!

bobhan1 commented Jul 25, 2024

Uh oh!

bobhan1 commented Jul 29, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dataroaring commented Aug 1, 2024

Uh oh!

zhannngchen left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Aug 1, 2024

Uh oh!

github-actionsBot commented Aug 1, 2024

Uh oh!

bobhan1 commented Aug 1, 2024

Uh oh!

zhannngchen left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Aug 2, 2024

Uh oh!

hust-hhb left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

bobhan1 commented Jul 24, 2024•
edited
Loading