Skip to content

[Bug] Avoid compacting recently added rowset - #3271

Merged
imay merged 1 commit into
apache:masterfrom
gaodayue:compaction_skip_window_pr
Apr 8, 2020
Merged

[Bug] Avoid compacting recently added rowset#3271
imay merged 1 commit into
apache:masterfrom
gaodayue:compaction_skip_window_pr

Conversation

@gaodayue

Copy link
Copy Markdown
Contributor

This CL fixes#3270 by skipping recently added version when performing cumulative compaction. A new config named "cumulative_compaction_skip_window_seconds" is added to adjust the time window.

@gaodayuegaodayue added kind/fix Categorizes issue or PR as related to a bug. area/storage Issues or PRs related to storage engine labels Apr 7, 2020
Comment threadbe/src/olap/rowset/rowset.cpp Outdated
@gaodayuegaodayue closed this Apr 7, 2020
@gaodayuegaodayue reopened this Apr 7, 2020
@imay

imay commented Apr 8, 2020

Copy link
Copy Markdown
Contributor

Hi, @gaodayue
After #3264 is merged, backend's configure can be changed without restart. Better to make cumulative_compaction_skip_window_seconds modifiable.

@gaodayue

Copy link
Copy Markdown
ContributorAuthor

@imay that's great, let me rebase

@gaodayue
gaodayueforce-pushed the compaction_skip_window_pr branch from 64e1cb0 to 57befd2CompareApril 8, 2020 04:07
imay
imay approved these changes Apr 8, 2020

@imayimay left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@imay

imay commented Apr 8, 2020

Copy link
Copy Markdown
Contributor

I have run this UT manually, and it is successful.
image
So, I will merge this PR.

@imay
imay merged commit 3557b12 into apache:masterApr 8, 2020
acelyc111 pushed a commit to acelyc111/incubator-doris that referenced this pull request Jan 20, 2021
This CL fixesapache#3270 by skipping recently added version when performing cumulative compaction. A new config named "cumulative_compaction_skip_window_seconds" is added to adjust the time window.
imay pushed a commit to StarRocks/starrocks that referenced this pull request Sep 21, 2022
…conds (#11490)
cumulative_compaction_skip_window_seconds introduced by apache/doris#3271. At that time, it can not acquire rowset version after compaction, but now we can acquire from _stale_rs_version_map.
so that this config is useless now.
mergifyBot pushed a commit to StarRocks/starrocks that referenced this pull request Oct 24, 2022
…conds (#11490)
cumulative_compaction_skip_window_seconds introduced by apache/doris#3271. At that time, it can not acquire rowset version after compaction, but now we can acquire from _stale_rs_version_map.
so that this config is useless now.
(cherry picked from commit 22af105)
# Conflicts:
#	be/src/storage/base_and_cumulative_compaction_policy.cpp
mergifyBot pushed a commit to StarRocks/starrocks that referenced this pull request Oct 24, 2022
…conds (#11490)
cumulative_compaction_skip_window_seconds introduced by apache/doris#3271. At that time, it can not acquire rowset version after compaction, but now we can acquire from _stale_rs_version_map.
so that this config is useless now.
(cherry picked from commit 22af105)
# Conflicts:
#	be/src/storage/base_and_cumulative_compaction_policy.cpp
mergifyBot pushed a commit to StarRocks/starrocks that referenced this pull request Oct 24, 2022
…conds (#11490)
cumulative_compaction_skip_window_seconds introduced by apache/doris#3271. At that time, it can not acquire rowset version after compaction, but now we can acquire from _stale_rs_version_map.
so that this config is useless now.
(cherry picked from commit 22af105)
# Conflicts:
#	be/src/storage/base_and_cumulative_compaction_policy.cpp
#	be/src/storage/cumulative_compaction.cpp
#	be/test/storage/cumulative_compaction_test.cpp
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/storageIssues or PRs related to storage enginekind/fixCategorizes issue or PR as related to a bug.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Query failed by "failed to initialize storage reader, res=-230"

2 participants

@gaodayue@imay