Skip to content

[improvement](show) Support that user can use show data skew statement instead of admin#7914

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
caiconghui merged 3 commits into from
Jan 29, 2022
Merged

[improvement](show) Support that user can use show data skew statement instead of admin #7914

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a601970
[improvement](show) Support that user can use show data skew statemen…
Jan 27, 2022
ef615d2
fix ut failed
Jan 27, 2022
89f6cb3
fix doc
Jan 28, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -62,6 +62,11 @@ ui/node_modules
ui/package-lock.json
docs/package-lock.json
fe/fe-common/.classpath
fe/.flattened-pom.xml
fe/fe-common/.flattened-pom.xml
fe/fe-core/.flattened-pom.xml
fe/spark-dpp/.flattened-pom.xml

rpc_data/

## ignore sample target
Expand Down
2 changes: 1 addition & 1 deletion docs/.vuepress/sidebar/en.js
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -524,7 +524,6 @@ module.exports = [
"ADMIN SHOW CONFIG",
"ADMIN SHOW REPLICA DISTRIBUTION",
"ADMIN SHOW REPLICA STATUS",
"ADMIN-SHOW-DATA-SKEW",
"ALTER CLUSTER",
"ALTER SYSTEM",
"CANCEL DECOMMISSION",
Expand DownExpand Up@@ -616,6 +615,7 @@ module.exports = [
"SHOW CREATE FUNCTION",
"SHOW CREATE ROUTINE LOAD",
"SHOW DATA",
"SHOW DATA SKEW",

@EmmyMiao87EmmyMiao87Jan 28, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This name must be same as the name of document ~

@caiconghuicaiconghuiJan 28, 2022

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

"SHOW DATABASES",
"SHOW DELETE",
"SHOW DYNAMIC PARTITION TABLES",
Expand Down
2 changes: 1 addition & 1 deletion docs/.vuepress/sidebar/zh-CN.js
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -528,7 +528,6 @@ module.exports = [
"ADMIN SHOW CONFIG",
"ADMIN SHOW REPLICA DISTRIBUTION",
"ADMIN SHOW REPLICA STATUS",
"ADMIN-SHOW-DATA-SKEW",
"ALTER CLUSTER",
"ALTER SYSTEM",
"CANCEL DECOMMISSION",
Expand DownExpand Up@@ -618,6 +617,7 @@ module.exports = [
"SHOW CREATE FUNCTION",
"SHOW CREATE ROUTINE LOAD",
"SHOW DATA",
"SHOW DATA SKEW",
"SHOW DATABASES",
"SHOW DELETE",
"SHOW DYNAMIC PARTITION TABLES",
Expand Down
10 changes: 5 additions & 5 deletions ...ts/Administration/ADMIN-SHOW-DATA-SKEW.md → ...ments/Data Manipulation/SHOW DATA SKEW.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
---
{
"title": "ADMIN SHOW DATA SKEW",
"title": "SHOW DATA SKEW",
"language": "en"
}
---
Expand All@@ -24,14 +24,14 @@ specific language governing permissions and limitations
under the License.
-->

# ADMIN SHOW DATA SKEW
# SHOW DATA SKEW
## description

This statement is used to view the data skew of a table or a partition.

grammar:

ADMIN SHOW DATA SKEW FROM [db_name.]tbl_name [PARTITION (p1)];
SHOW DATA SKEW FROM [db_name.]tbl_name [PARTITION (p1)];

Description:

Expand All@@ -42,9 +42,9 @@ under the License.

1. View the data skew of the table

ADMIN SHOW DATA SKEW FROM db1.test PARTITION(p1);
SHOW DATA SKEW FROM db1.test PARTITION(p1);

## keyword

ADMIN, SHOW, DATA, SKEW
SHOW, DATA, SKEW

10 changes: 5 additions & 5 deletions ...ts/Administration/ADMIN-SHOW-DATA-SKEW.md → ...ments/Data Manipulation/SHOW DATA SKEW.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
---
{
"title": "ADMIN SHOW DATA SKEW",
"title": "SHOW DATA SKEW",
"language": "zh-CN"
}
---
Expand All@@ -24,14 +24,14 @@ specific language governing permissions and limitations
under the License.
-->

# ADMIN SHOW DATA SKEW
# SHOW DATA SKEW
## description

该语句用于查看表或某个分区的数据倾斜情况。

语法：

ADMIN SHOW DATA SKEW FROM [db_name.]tbl_name PARTITION (partition_name);
SHOW DATA SKEW FROM [db_name.]tbl_name PARTITION (partition_name);

说明：

Expand All@@ -42,9 +42,9 @@ under the License.

1. 查看表的数据倾斜情况

ADMIN SHOW DATA SKEW FROM db1.test PARTITION(p1);
SHOW DATA SKEW FROM db1.test PARTITION(p1);

## keyword

ADMIN,SHOW,DATA,SKEW
SHOW,DATA,SKEW

8 changes: 4 additions & 4 deletions fe/fe-core/src/main/cup/sql_parser.cup
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -2717,6 +2717,10 @@ show_param ::=
{:
RESULT = new ShowAlterStmt(type, db, parser.where, orderByClause, limitClause);
:}
| KW_SHOW KW_DATA KW_SKEW KW_FROM base_table_ref:table_ref
{:
RESULT = new ShowDataSkewStmt(table_ref);
:}
/* Show data statement: used to show data size of specified range */
| KW_DATA order_by_clause:orderByClause
{:
Expand DownExpand Up@@ -5253,10 +5257,6 @@ admin_stmt ::=
{:
RESULT = new AdminCheckTabletsStmt(tabletIds, properties);
:}
| KW_ADMIN KW_SHOW KW_DATA KW_SKEW KW_FROM base_table_ref:table_ref
{:
RESULT = new AdminShowDataSkewStmt(table_ref);
:}
| KW_ADMIN KW_CLEAN KW_TRASH KW_ON LPAREN string_list:backends RPAREN
{:
RESULT = new AdminCleanTrashStmt(backends);
Expand Down
1 change: 0 additions & 1 deletion fe/fe-core/src/main/java/org/apache/doris/analysis/CreateDbStmt.java
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -32,7 +32,6 @@
import java.util.HashMap;
import java.util.Map;

// 用于描述CREATE DATABASE的内部结构
public class CreateDbStmt extends DdlStmt {
private boolean ifNotExists;
private String dbName;
Expand Down
21 changes: 11 additions & 10 deletions ...doris/analysis/AdminShowDataSkewStmt.java → ...ache/doris/analysis/ShowDataSkewStmt.java
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -30,30 +30,31 @@

import com.google.common.collect.ImmutableList;

// admin show data skew from tbl [partition(p1, p2, ...)]
public class AdminShowDataSkewStmt extends ShowStmt {
// show data skew from tbl [partition(p1, p2, ...)]
public class ShowDataSkewStmt extends ShowStmt {
public static final ImmutableList<String> TITLE_NAMES = new ImmutableList.Builder<String>()
.add("BucketIdx").add("AvgDataSize")
.add("Graph").add("Percent")
.build();

private TableRef tblRef;

public AdminShowDataSkewStmt(TableRef tblRef) {
public ShowDataSkewStmt(TableRef tblRef) {
this.tblRef = tblRef;
}

@Override
public void analyze(Analyzer analyzer) throws UserException {
super.analyze(analyzer);

// check auth
if (!Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(ConnectContext.get(), PrivPredicate.ADMIN)) {
ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR, "ADMIN");
}

tblRef.getName().analyze(analyzer);

if (!Catalog.getCurrentCatalog().getAuth().checkTblPriv(ConnectContext.get(), tblRef.getName().getDb(),
tblRef.getName().getTbl(),
PrivPredicate.SHOW)) {
ErrorReport.reportAnalysisException(ErrorCode.ERR_TABLEACCESS_DENIED_ERROR, "SHOW DATA SKEW",
ConnectContext.get().getQualifiedUser(),
ConnectContext.get().getRemoteIP(),
tblRef.getName().getDb() + "." + tblRef.getName().getTbl());
}
PartitionNames partitionNames = tblRef.getPartitionNames();
if (partitionNames == null || partitionNames.getPartitionNames().size() != 1) {
throw new AnalysisException("Should specify one and only one partition");
Expand Down
7 changes: 3 additions & 4 deletions fe/fe-core/src/main/java/org/apache/doris/catalog/MetadataViewer.java
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -17,7 +17,7 @@

package org.apache.doris.catalog;

import org.apache.doris.analysis.AdminShowDataSkewStmt;
import org.apache.doris.analysis.ShowDataSkewStmt;
import org.apache.doris.analysis.AdminShowReplicaDistributionStmt;
import org.apache.doris.analysis.AdminShowReplicaStatusStmt;
import org.apache.doris.analysis.BinaryPredicate.Operator;
Expand DownExpand Up@@ -247,7 +247,7 @@ private static String graph(long num, long totalNum) {
return sb.toString();
}

public static List<List<String>> getDataSkew(AdminShowDataSkewStmt stmt) throws DdlException {
public static List<List<String>> getDataSkew(ShowDataSkewStmt stmt) throws DdlException {
return getDataSkew(stmt.getDbName(), stmt.getTblName(), stmt.getPartitionNames());
}

Expand All@@ -257,7 +257,6 @@ private static List<List<String>> getDataSkew(String dbName, String tblName, Par

List<List<String>> result = Lists.newArrayList();
Catalog catalog = Catalog.getCurrentCatalog();
SystemInfoService infoService = Catalog.getCurrentSystemInfo();

if (partitionNames == null || partitionNames.getPartitionNames().size() != 1) {
throw new DdlException("Should specify one and only one partitions");
Expand DownExpand Up@@ -288,7 +287,7 @@ private static List<List<String>> getDataSkew(String dbName, String tblName, Par
List<Long> tabletIds = mIndex.getTabletIdsInOrder();
for (int i = 0; i < tabletIds.size(); i++) {
Tablet tablet = mIndex.getTablet(tabletIds.get(i));
long dataSize = tablet.getDataSize(false);
long dataSize = tablet.getDataSize(true);

@EmmyMiao87EmmyMiao87Jan 27, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe single replica is better ?

@caiconghuicaiconghuiJan 27, 2022

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

now，just according to the average data size column，here should compute average data size

tabletInfos.set(i, tabletInfos.get(i) + dataSize);
totalSize += dataSize;
}
Expand Down
8 changes: 1 addition & 7 deletions fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -382,13 +382,7 @@ public boolean equals(Object obj) {
public long getDataSize(boolean singleReplica) {
LongStream s = replicas.stream().filter(r -> r.getState() == ReplicaState.NORMAL)
.mapToLong(Replica::getDataSize);
return singleReplica ? Double.valueOf(s.average().getAsDouble()).longValue() : s.sum();
}

public long getRowNum(boolean singleReplica) {
LongStream s = replicas.stream().filter(r -> r.getState() == ReplicaState.NORMAL)
.mapToLong(Replica::getRowCount);
return singleReplica ? Double.valueOf(s.average().getAsDouble()).longValue() : s.sum();
return singleReplica ? Double.valueOf(s.average().orElse(0)).longValue() : s.sum();
}

/**
Expand Down
10 changes: 5 additions & 5 deletions fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -18,7 +18,6 @@
package org.apache.doris.qe;

import org.apache.doris.analysis.AdminShowConfigStmt;
import org.apache.doris.analysis.AdminShowDataSkewStmt;
import org.apache.doris.analysis.AdminShowReplicaDistributionStmt;
import org.apache.doris.analysis.AdminShowReplicaStatusStmt;
import org.apache.doris.analysis.DescribeStmt;
Expand All@@ -37,6 +36,7 @@
import org.apache.doris.analysis.ShowCreateFunctionStmt;
import org.apache.doris.analysis.ShowCreateRoutineLoadStmt;
import org.apache.doris.analysis.ShowCreateTableStmt;
import org.apache.doris.analysis.ShowDataSkewStmt;
import org.apache.doris.analysis.ShowDataStmt;
import org.apache.doris.analysis.ShowDbIdStmt;
import org.apache.doris.analysis.ShowDbStmt;
Expand DownExpand Up@@ -317,8 +317,8 @@ public ShowResultSet execute() throws AnalysisException {
handleShowQueryProfile();
} else if (stmt instanceof ShowLoadProfileStmt) {
handleShowLoadProfile();
} else if (stmt instanceof AdminShowDataSkewStmt) {
handleAdminShowDataSkew();
} else if (stmt instanceof ShowDataSkewStmt) {
handleShowDataSkew();
} else if (stmt instanceof ShowSyncJobStmt) {
handleShowSyncJobs();
} else if (stmt instanceof ShowSqlBlockRuleStmt) {
Expand DownExpand Up@@ -2026,8 +2026,8 @@ private void handleShowCreateRoutineLoad() throws AnalysisException {
resultSet = new ShowResultSet(showCreateRoutineLoadStmt.getMetaData(), rows);
}

private void handleAdminShowDataSkew() throws AnalysisException {
AdminShowDataSkewStmt showStmt = (AdminShowDataSkewStmt) stmt;
private void handleShowDataSkew() throws AnalysisException {
ShowDataSkewStmt showStmt = (ShowDataSkewStmt) stmt;
try {
List<List<String>> results = MetadataViewer.getDataSkew(showStmt);
resultSet = new ShowResultSet(showStmt.getMetaData(), results);
Expand Down
4 changes: 2 additions & 2 deletions fe/fe-core/src/test/java/org/apache/doris/analysis/AdminShowReplicaTest.java
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -92,8 +92,8 @@ public void testShowReplicaDistribution() throws Exception {
Assert.assertEquals(1, resultSet.getResultRows().size());
Assert.assertEquals(7, resultSet.getResultRows().get(0).size());

stmtStr = "admin show data skew from test.tbl1 partition(p1)";
AdminShowDataSkewStmt skewStmt = (AdminShowDataSkewStmt) UtFrameUtils.parseAndAnalyzeStmt(
stmtStr = "show data skew from test.tbl1 partition(p1)";
ShowDataSkewStmt skewStmt = (ShowDataSkewStmt) UtFrameUtils.parseAndAnalyzeStmt(
stmtStr, connectContext);
executor = new ShowExecutor(connectContext, skewStmt);
resultSet = executor.execute();
Expand Down
8 changes: 4 additions & 4 deletions fe/fe-core/src/test/java/org/apache/doris/analysis/CreateDbStmtTest.java
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -48,15 +48,15 @@ public void setUp() {
}

@Test
public void testAnalyzeNormal() throws UserException, AnalysisException {
public void testAnalyzeNormal() throws UserException {
CreateDbStmt dbStmt = new CreateDbStmt(false, "test", null);
dbStmt.analyze(analyzer);
Assert.assertEquals("testCluster:test", dbStmt.getFullDbName());
Assert.assertEquals("CREATE DATABASE `testCluster:test`", dbStmt.toString());
}

@Test(expected = AnalysisException.class)
public void testAnalyzeWithException() throws UserException, AnalysisException {
public void testAnalyzeWithException() throws UserException {
CreateDbStmt stmt = new CreateDbStmt(false, "", null);
stmt.analyze(analyzer);
Assert.fail("no exception");
Expand All@@ -82,8 +82,8 @@ public void testAnalyzeIcebergWithException() throws UserException {
Map<String, String> properties = new HashMap<>();
properties.put("iceberg.database", "doris");
properties.put("iceberg.hive.metastore.uris", "thrift://127.0.0.1:9087");
CreateDbStmt stmt = new CreateDbStmt(false, "test", properties);
CreateDbStmt stmt = new CreateDbStmt(false, "", properties);
stmt.analyze(analyzer);
Assert.fail("no exception");
Assert.fail("No exception throws.");
}
}
7 changes: 4 additions & 3 deletions fe/fe-core/src/test/java/org/apache/doris/analysis/ShowTableCreationStmtTest.java
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -46,9 +46,10 @@ public void testNormal() throws UserException {
ShowTableCreationStmt stmt = new ShowTableCreationStmt("doris", "log");
stmt.analyze(analyzer);
Assert.assertEquals("SHOW TABLE CREATION FROM `testCluster:doris` LIKE `log`", stmt.toString());
Assert.assertEquals(4, stmt.getMetaData().getColumnCount());
Assert.assertEquals("Table", stmt.getMetaData().getColumn(0).getName());
Assert.assertEquals("Status", stmt.getMetaData().getColumn(1).getName());
Assert.assertEquals(5, stmt.getMetaData().getColumnCount());
Assert.assertEquals("Database", stmt.getMetaData().getColumn(0).getName());
Assert.assertEquals("Table", stmt.getMetaData().getColumn(1).getName());
Assert.assertEquals("Status", stmt.getMetaData().getColumn(2).getName());
}

@Test
Expand Down
Loading