Uh oh!
There was an error while loading. Please reload this page.
HBASE-22012 Space Quota: DisableTableViolationPolicy will cause cycles of enable/disable table - #572
Closed
shardul-cr7 wants to merge 3 commits into
Closed
HBASE-22012 Space Quota: DisableTableViolationPolicy will cause cycles of enable/disable table#572shardul-cr7 wants to merge 3 commits into
shardul-cr7 wants to merge 3 commits into
Conversation
Apache-HBase
commented
Sep 1, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Sep 2, 2019
💔 -1 overall
This message was automatically generated. |
saintstack
left a comment
Contributor
There was a problem hiding this comment.
Skimmed. Makes sense to me.
Apache-HBase
commented
Sep 2, 2019
💔 -1 overall
This message was automatically generated. |
saintstack
left a comment
Contributor
There was a problem hiding this comment.
@joshelser Mind taking a look at this one sir?
joshelser
commented
Sep 16, 2019
Member
Yep. It's on my list -- Shardul pinged me already :) Thanks for asking, S. |
joshelser
requested changes
Sep 18, 2019
Member
There was a problem hiding this comment.
Approach looks OK to solve this issue. I'm a little worried as this is holding on to objects in memory for a potentially long time, but I can't think of a different way to get around that easily.
I think this approach will break down after a restart of the Master. RegionSize reports are kept in memory, but after a table is disabled, we'll no longer get those reports coming into the Master. Thus, we'll flip the table back to active because we think we're missing reports and we have no way to re-create the old RegionSizes we had. This might be some follow-on work to do rather than take this back to the drawing board :)
| long currentEntryTime = regionSizes.get(regionInfo).getTime(); | ||
| boolean isInViolationAndPolicyDisable = isInViolationAndPolicyDisable(regionInfo.getTable()); | ||
| // do not prune the entries if table is in violation and | ||
| // violation policy is disable.prune entries older than time. |
Member
There was a problem hiding this comment.
Could you buff out this comment with an explanation as to why we need to hold onto RegionSizes for tables in violation with a disable policy or include a reference to HBASE-22012, please?
| boolean isInViolationAndPolicyDisable = false; | ||
| SpaceViolationPolicy policy = null; | ||
| try { | ||
| if (QuotaUtil.isQuotaEnabled(masterServices.getConfiguration())) { |
Member
There was a problem hiding this comment.
I think this is unnecessary to re-check.
QuotaObserverChore would be calling pruneEntriesOlderThan(long) but that chore is only scheduled if quotas are enabled.
| if (QuotaUtil.isQuotaEnabled(masterServices.getConfiguration())) { | ||
| // Get Current Snapshot for the given table | ||
| SpaceQuotaSnapshot spaceQuotaSnapshot = | ||
| QuotaUtil.getCurrentSnapshotFromQuotaTable(masterServices.getConnection(), tableName); |
Member
There was a problem hiding this comment.
Can you use QuotaObserverChore#getTableQuotaSnapshot(TableName) instead? This is going to hit hbase:quota every time which might be costly.
Since we're inside the master already, we should be able to safely pull from the in-memory state in QuotaObserverChore instead.
ContributorAuthor
There was a problem hiding this comment.
using QuotaObserverChore#getTableQuotaSnapshot(TableName) now.
| if (spaceQuotaSnapshot != null) { | ||
| // check if table in violation | ||
| isInViolationAtTable = spaceQuotaSnapshot.getQuotaStatus().isInViolation(); | ||
| Optional<SpaceViolationPolicy> policyAtNamespace = |
Member
There was a problem hiding this comment.
nit: just policy? AtNamespace doesn't seem to be appropriate.
| } | ||
| } | ||
| } | ||
| isInViolationAndPolicyDisable = |
Member
There was a problem hiding this comment.
can just make this return policy == SpaceViolationPolicy.DISABLE && isInViolationAtTable;
Member
There was a problem hiding this comment.
If you do this, then you can also remove the variable isInViolationAndPolicyDisable completely, lift the return isInViolationAndPolicyDisable into the catch block and make it return false instead.
| HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); | ||
| Configuration conf = TEST_UTIL.getConfiguration(); | ||
| conf.set(QUOTA_CONF_KEY, "false"); | ||
| when(masterServices.getConfiguration()).thenReturn(conf); |
Member
There was a problem hiding this comment.
Why are you disabling the quota feature here?
| public void testDisablePolicyQuotaAndViolate() throws Exception { | ||
| TableName tableName = helper.createTable(); | ||
| helper.setQuotaLimit(tableName, SpaceViolationPolicy.DISABLE, 2L); | ||
| helper.writeData(tableName, SpaceQuotaHelperForTests.ONE_MEGABYTE * 3L); |
Member
There was a problem hiding this comment.
1MB should be enough to trigger the quota, no?
| MasterQuotaManager quotaManager = master.getMasterQuotaManager(); | ||
| // Sufficient time for all the chores to run. | ||
| Thread.sleep(5000); |
Member
There was a problem hiding this comment.
What is the state you're actually expecting to happen? RegionSize reports to be sent in?
Same issue as one of your recent PRs. Can you please convert this into an wait+explicit check?
ContributorAuthor
There was a problem hiding this comment.
done. added a wait predicate.
| // Check if disabled table region report present in the map after retention period expired. | ||
| // It should be present after retention period expired. | ||
| for (Map.Entry<RegionInfo, Long> entry : quotaManager.snapshotRegionSizes().entrySet()) { |
Member
There was a problem hiding this comment.
What about making this whole for-loop:
quotaManager.snapshotRegionSize().keySet()
.stream()
.filter(k -> k.getTable().equals(tableName))
.count() > 0`
shardul-cr7
commented
Sep 21, 2019
ContributorAuthor
@joshelser , thanks for the review..did all the changes and pushed. |
Apache-HBase
commented
Sep 22, 2019
💔 -1 overall
This message was automatically generated. |
joshelser
requested changes
Sep 22, 2019
joshelser
left a comment
Member
There was a problem hiding this comment.
One bug in the latest patch (thanks findbugs) and one minor readability request. Otherwise, I think this is fine.
| // check namespace in violation | ||
| isInViolationAtNamespace = namespaceQuotaSnapshot.getQuotaStatus().isInViolation(); | ||
| Optional<SpaceViolationPolicy> namespacePolicy = | ||
| tableQuotaSnapshot.getQuotaStatus().getPolicy(); |
Member
There was a problem hiding this comment.
This is the NPE from findbugs. Should be namespaceQuotaSnapshot.
Uh oh!
There was an error while loading. Please reload this page.
| // Check if disabled table region report present in the map after retention period expired. | ||
| // It should be present after retention period expired. | ||
| Assert.assertTrue(quotaManager.snapshotRegionSizes().keySet().stream() |
Member
There was a problem hiding this comment.
Can you pull the lambda out of the assertTrue to make this a little more readable? e.g.
final int regionSizes = quotaManager.snapshotRegionSizes()...;
Assert.assertTrue(regionSizes > 0);
Apache-HBase
commented
Sep 23, 2019
💔 -1 overall
This message was automatically generated. |
joshelser
commented
Sep 25, 2019
Member
Looks ok now. Let me try to get this in. |
joshelser
commented
Sep 26, 2019
Member
Required some minor changes across branch-2.2/2 and more changes in branch-2.1. Re-running the UTs on branch-2.1 now. |
asfgit pushed a commit
that referenced
this pull request
Sep 26, 2019
…lation policy Space quotas has a feature which intends to avoid enacting a space quota violation policy when only a subset of the Regions for that Table have reported their space usage (under the assumption that we cannot make an informed decision if we do not include all regions in our calculations). This had the unintended side-effect, when a table is disabled as a part of a violation policy, of causing the regions for that table to not be reported which disables the violation policy and enables the table. Need to make sure that when a table is disabled because of a violation policy that the code does not automatically move that table out of violation because region sizes are not being reported (because those regions are not open). Closes#572 Signed-off-by: Josh Elser <elserj@apache.org>
asfgit pushed a commit
that referenced
this pull request
Sep 26, 2019
…lation policy Space quotas has a feature which intends to avoid enacting a space quota violation policy when only a subset of the Regions for that Table have reported their space usage (under the assumption that we cannot make an informed decision if we do not include all regions in our calculations). This had the unintended side-effect, when a table is disabled as a part of a violation policy, of causing the regions for that table to not be reported which disables the violation policy and enables the table. Need to make sure that when a table is disabled because of a violation policy that the code does not automatically move that table out of violation because region sizes are not being reported (because those regions are not open). elserj: Had to replace some usages of Optional that don't exist in branch-2.1 Closes#572 Signed-off-by: Josh Elser <elserj@apache.org>
asfgit pushed a commit
that referenced
this pull request
Sep 26, 2019
…lation policy Space quotas has a feature which intends to avoid enacting a space quota violation policy when only a subset of the Regions for that Table have reported their space usage (under the assumption that we cannot make an informed decision if we do not include all regions in our calculations). This had the unintended side-effect, when a table is disabled as a part of a violation policy, of causing the regions for that table to not be reported which disables the violation policy and enables the table. Need to make sure that when a table is disabled because of a violation policy that the code does not automatically move that table out of violation because region sizes are not being reported (because those regions are not open). Closes#572 Signed-off-by: Josh Elser <elserj@apache.org>
infraio pushed a commit
to infraio/hbase
that referenced
this pull request
Aug 17, 2020
…lation policy Space quotas has a feature which intends to avoid enacting a space quota violation policy when only a subset of the Regions for that Table have reported their space usage (under the assumption that we cannot make an informed decision if we do not include all regions in our calculations). This had the unintended side-effect, when a table is disabled as a part of a violation policy, of causing the regions for that table to not be reported which disables the violation policy and enables the table. Need to make sure that when a table is disabled because of a violation policy that the code does not automatically move that table out of violation because region sizes are not being reported (because those regions are not open). Closesapache#572 Signed-off-by: Josh Elser <elserj@apache.org>
symat pushed a commit
to symat/hbase
that referenced
this pull request
Feb 17, 2021
…lation policy Space quotas has a feature which intends to avoid enacting a space quota violation policy when only a subset of the Regions for that Table have reported their space usage (under the assumption that we cannot make an informed decision if we do not include all regions in our calculations). This had the unintended side-effect, when a table is disabled as a part of a violation policy, of causing the regions for that table to not be reported which disables the violation policy and enables the table. Need to make sure that when a table is disabled because of a violation policy that the code does not automatically move that table out of violation because region sizes are not being reported (because those regions are not open). Closesapache#572 Signed-off-by: Josh Elser <elserj@apache.org> (cherry picked from commit 16d6a9f) Change-Id: I01cbb39ef7046fedf2feff558b9f414b6397a19e
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Space Quota: Policy state is getting changed from disable to Observance after sometime automatically.
Steps:
1: Create a table with space quota policy as Disable
2: Put some data so that table state is in space quota violation
3: So observe that table state is in violation
4: Now wait for some time
5: Observe that after some time table state is changing to to Observance however table is still disabled
Solution: it is better to not remove the entries from cache regionSizes if table is in violation and has DISABLE policy set on it to avoid this repeat cycle of enable/disable. If the entries are not removed from the cache for disable policy and violation tables the table will not go from disable to enable even though regions of the table are offline.