Add delayed deletion of rowsets function, fix -230 error. by ZhangYu0123 · Pull Request #4039 · apache/doris

ZhangYu0123 · 2020-07-07T08:51:52Z

Related issue #4017, main changes as follows:

Add expired_snapshot_rs_version_map，_expired_snapshot_rs_metas，
Add VersionedRowsetTracker record compacted path version
Record path version when rowsets compact
In gc process, add expired snapshot rowsets to unused set to remove.

wuyunfeng · 2020-07-07T08:56:42Z

- _rs_graph.construct_rowset_graph(_tablet_meta->all_rs_metas());
+ // change _rs_graph to _versioned_rs_tracker
+ // _rs_graph.construct_rowset_graph(_tablet_meta->all_rs_metas());
+ _versioned_rs_tracker.construct_versioned_tracker(_tablet_meta->all_rs_metas(),


66 - 67 has nonstandard format

morningman · 2020-07-07T13:43:05Z

- RETURN_NOT_OK(gc_unused_rowsets());
- TRACE("unused rowsets have been moved to GC queue");
+// RETURN_NOT_OK(gc_unused_rowsets());
+// TRACE("unused rowsets have been moved to GC queue");


morningman · 2020-07-07T13:45:27Z

+}
+
+void RowsetGraph::construct_rowset_graph(const std::vector<RowsetMetaSharedPtr>& rs_metas,
+ int64_t& max_version) {


Suggested change
int64_t& max_version) {
int64_t* max_version) {
Better to use point to indicate it is a output parameter.

morningman · 2020-07-07T13:50:08Z

-
+/// RowsetGraph class which is implemented to build and maintain total versions of rowsets. 
+/// This class use adjacency-matrix represent rowsets version and links. A vertex is a version 
+/// and a link is the _version object of a rowset (from start version to end version).


Suggested change
/// and a link is the _version object of a rowset (from start version to end version).
/// and a link is the _version object of a rowset (from start version to end version + 1).

morningman · 2020-07-07T13:52:59Z

 };

-} // namespace doris
+/// VersionTracker class which is implemented to maintain compacted version path of rowsets. 


Suggested change
/// VersionTracker class which is implemented to maintain compacted version path of rowsets.
/// VersionTracker class which is implemented to maintain multi-version path of rowsets.

morningman · 2020-07-07T14:10:55Z

+/// VersionedRowsetTracker class is responsible to track all rowsets version links of a tablet.
+/// This class not only records the graph of all versions, but also records the paths which will be removed
+/// after the path is expired.
+class VersionedRowsetTracker {


Suggested change
classVersionedRowsetTracker {
classTimestampedVersionTracker {

morningman · 2020-07-07T14:21:54Z

+ // 2 remove this path from other_path
+ auto path_iter = path_version_ptr->begin();
+ for (; path_iter != path_version_ptr->end(); path_iter++) {
+ Version version = (*path_iter)->version();


Suggested change
Version version = (*path_iter)->version();
constVersion& version = (*path_iter)->version();

morningman · 2020-07-07T14:24:02Z

+ _construct_versioned_tracker(rs_metas, expired_snapshot_rs_metas);
+}
+
+void VersionedRowsetTracker::reconstruct_versioned_tracker(


Looks same as construct_versioned_tracker()

morningman · 2020-07-07T14:26:12Z

+
+ while (iter != _expired_snapshot_rs_path_map.end()) {
+ std::vector<VersionTrackerSharedPtr>::iterator version_path_iter = iter->second->begin();
+ int64_t max_create_time = -1;


I think we can save the max_create_time of a version path in _expired_snapshot_rs_path_map

morningman · 2020-07-07T14:57:04Z

+ }
+}
+
+void VersionedRowsetTracker::_print_current_state() {


Better to return a string, and let the caller decide how to print it.

morningman

LGTM

morningman

LGTM

wuyunfeng requested changes Jul 7, 2020
View reviewed changes

xy720 reviewed Jul 7, 2020
View reviewed changes

Comment threadbe/src/olap/rowset_graph.h Outdated

ZhangYu0123 force-pushed the merged_rowset_delay_unused branch from 0f8d14d to 5eb43d3Compare July 7, 2020 12:17

morningman added area/storage Issues or PRs related to storage engine kind/improvement labels Jul 7, 2020

morningman reviewed Jul 7, 2020
View reviewed changes

wutiangan reviewed Jul 14, 2020
View reviewed changes

Comment threadbe/src/olap/tablet.cpp Outdated

morningman reviewed Jul 14, 2020
View reviewed changes

Comment threadbe/src/olap/tablet_meta.h Outdated
Comment threadbe/src/olap/tablet.cpp Outdated

morningman reviewed Jul 15, 2020
View reviewed changes

Comment threadbe/src/olap/tablet.cpp Outdated
Comment threadbe/src/olap/tablet.cpp Outdated
Comment threadbe/src/olap/tablet.cpp Outdated
Comment threadbe/src/olap/tablet.cpp Outdated
Comment threadbe/src/olap/tablet_meta.cpp Outdated

ZhangYu0123 added 11 commits July 15, 2020 23:44

merged rowset version delay unused, fix -230 error
68a34aa

Add delayed deletion of rowsets function, fix -230 error.
ecf54c2

Add delayed deletion of rowsets function, fix -230 error.
7c8489a

Add delayed deletion of rowsets function, fix -230 error.
a638c7e

Add expired path version json info
b8f23aa

Add delayed deletion of rowsets function, fix -230 error.
1dd0e78

Add delayed deletion of rowsets function, fix -230 error.
05080a2

Add delayed deletion of rowsets function, fix -230 error.
2aa7693

Add delayed deletion of rowsets function, fix -230 error.
f9b59cb

Add delayed deletion of rowsets function, fix -230 error.
24a8ff5

Add delayed deletion of rowsets function, fix -230 error.
64713d0

ZhangYu0123 force-pushed the merged_rowset_delay_unused branch from 3ed0620 to 64713d0Compare July 15, 2020 16:56

Add delayed deletion of rowsets function, fix -230 error.
dbed890

morningman approved these changes Jul 17, 2020
View reviewed changes

morningman added the approved Indicates a PR has been approved by one committer. label Jul 19, 2020

morningman merged commit 03cf9b2 into apache:masterJul 19, 2020

EmmyMiao87 mentioned this pull request Aug 17, 2020
Release Notes 0.13.0 #4370
Closed

csun5285 added a commit to csun5285/doris that referenced this pull request Sep 23, 2025
[enhance](variant) get compaction subcolumns type (apache#4039)
7b5ab6c

	/// and a link is the _version object of a rowset (from start version to end version).
	/// and a link is the _version object of a rowset (from start version to end version + 1).

	/// VersionTracker class which is implemented to maintain compacted version path of rowsets.
	/// VersionTracker class which is implemented to maintain multi-version path of rowsets.

	classVersionedRowsetTracker {
	classTimestampedVersionTracker {

	Version version = (*path_iter)->version();
	constVersion& version = (*path_iter)->version();

Uh oh!

Conversation

ZhangYu0123 commented Jul 7, 2020• edited by morningman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ZhangYu0123 commented Jul 7, 2020•
edited by morningman
Loading