Skip to content

Support setting replica quota in db level - #3283

Merged
morningman merged 9 commits into
apache:masterfrom
caiconghui:replica_quota
Apr 14, 2020
Merged

Support setting replica quota in db level#3283
morningman merged 9 commits into
apache:masterfrom
caiconghui:replica_quota

Conversation

@caiconghui

@caiconghuicaiconghui commented Apr 9, 2020

Copy link
Copy Markdown
Contributor

This PR is to limit the replica usage, admin need to know the replica usage for every db and table, be able to set replica quota for every db.

@imay

imay commented Apr 9, 2020

Copy link
Copy Markdown
Contributor

@caiconghui
Better to create an issue to track this feature.
And let others discuss about it.

@caiconghui

caiconghui commented Apr 9, 2020

Copy link
Copy Markdown
ContributorAuthor

@imay OK for #3285

@chaoylichaoyli added this to the 0.13.0 milestone Apr 9, 2020
morningman
morningman requested changes Apr 9, 2020
View reviewed changes
Comment threaddocs/documentation/cn/sql-reference/sql-statements/Data Definition/ALTER DATABASE.md
Comment threadfe/src/main/cup/sql_parser.cup Outdated
Comment threadfe/src/main/java/org/apache/doris/analysis/ShowDataStmt.java Outdated
Comment threadfe/src/main/java/org/apache/doris/catalog/Catalog.java Outdated
Comment threadfe/src/main/java/org/apache/doris/catalog/Catalog.java Outdated
fullNameToDb.put(newFullDbName, db);

DatabaseInfo dbInfo = new DatabaseInfo(fullDbName, newFullDbName, -1L);
DatabaseInfo dbInfo = new DatabaseInfo(fullDbName, newFullDbName, -1L, QuotaType.DATA);

@morningmanmorningmanApr 9, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about change QuotaType.DATA to null? For better understanding?

@caiconghuicaiconghuiApr 10, 2020

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@morningman null may cause NullPointerException, so I use QuotaType.NONE instead of null

Comment threadfe/src/main/java/org/apache/doris/catalog/Database.java
Comment threadfe/src/main/java/org/apache/doris/catalog/Database.java
Comment threadfe/src/main/java/org/apache/doris/common/proc/DbsProcDir.java Outdated
Comment threadfe/src/main/java/org/apache/doris/common/proc/TablesProcDir.java Outdated
Comment threadfe/src/main/java/org/apache/doris/persist/DatabaseInfo.java Outdated
caiconghui [蔡聪辉] added 2 commits April 10, 2020 13:19
kangkaisen
kangkaisen reviewed Apr 10, 2020
View reviewed changes
Comment threadfe/src/main/java/org/apache/doris/common/FeConstants.java Outdated
public static int shortkey_max_column_count = 3;
public static int shortkey_maxsize_bytes = 36;
public static long default_db_data_quota_bytes = 1024 * 1024 * 1024 * 1024L; // 1TB
public static long default_db_replica_quota_size = 1000000;

@kangkaisenkangkaisenApr 10, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 1000000 is too small. In meituan.com, the replica number for one database has exceeded 1000000. I think your initial value 1024 * 1024 * 1024 is OK.

@imayimayApr 10, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not set -1 as default value, which means no limit?
Otherwise I'm afraid some cluster can't create table after cluster upgrade.

@caiconghuicaiconghuiApr 10, 2020

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. I will change it to original default value

@caiconghuicaiconghuiApr 10, 2020

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@imay 1024 * 1024 * 1024 is a large value and enough, if one table has 100000 tablets , about 10000 tables can exits in on db?

@caiconghuicaiconghuiApr 10, 2020

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in my point of view, quota is a necessary limit for resource usage, the resource is limited, replica quota should like data quota, not set to be -1 @imay

kangkaisen
kangkaisen reviewed Apr 10, 2020
View reviewed changes
Comment threadfe/src/main/java/org/apache/doris/catalog/Database.java Outdated
morningman
morningman approved these changes Apr 11, 2020
View reviewed changes

@morningmanmorningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment threadfe/src/main/cup/sql_parser.cup Outdated
morningman
morningman requested changes Apr 11, 2020
View reviewed changes

@morningmanmorningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the syntax file

morningman
morningman approved these changes Apr 12, 2020
View reviewed changes

@morningmanmorningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman
morningman merged commit 9257535 into apache:masterApr 14, 2020
@caiconghui
caiconghui deleted the replica_quota branch May 25, 2020 03:08
@EmmyMiao87EmmyMiao87 mentioned this pull request Aug 17, 2020
Closed
acelyc111 pushed a commit to acelyc111/incubator-doris that referenced this pull request Jan 20, 2021
@caiconghui@WingsGo
[New Feature] Support setting replica quota in db level (apache#3283)
925b7e9
This PR is to limit the replica usage, admin need to know the replica usage for every db and table, be able to set replica quota for every db.
```
ALTER DATABASE db_name SET REPLICA QUOTA quota; ```
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@imayimayimay left review comments
@kangkaisenkangkaisenkangkaisen left review comments
@morningmanmorningmanmorningman approved these changes

Assignees

@caiconghuicaiconghui

Labels

kind/improvement

Projects

None yet

Milestone

0.13.0

Development

Successfully merging this pull request may close these issues.

5 participants

@caiconghui@imay@morningman@kangkaisen@chaoyli