[feature](information_schema)add metadata_name_ids for quickly get catlogs,db,table and add profiling table in order to Compatible with mysql by hubgeter · Pull Request #22702 · apache/doris

hubgeter · 2023-08-08T02:02:33Z

Proposed changes

add information_schema.metadata_name_idsfor quickly get catlogs,db,table.

table struct :

mysql>descinternal.information_schema.metadata_name_ids;
+---------------+--------------+------+-------+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------------+--------------+------+-------+---------+-------+
| CATALOG_ID | BIGINT | Yes | false | NULL | |
| CATALOG_NAME | VARCHAR(512) | Yes | false | NULL | |
| DATABASE_ID | BIGINT | Yes | false | NULL | |
| DATABASE_NAME | VARCHAR(64) | Yes | false | NULL | |
| TABLE_ID | BIGINT | Yes | false | NULL | |
| TABLE_NAME | VARCHAR(64) | Yes | false | NULL | |
+---------------+--------------+------+-------+---------+-------+6 rows inset (0.00 sec) mysql>select*frominternal.information_schema.metadata_name_ids where CATALOG_NAME="hive1"limit1 \G;
***************************1. row ***************************
CATALOG_ID: 113008
CATALOG_NAME: hive1
DATABASE_ID: 113042
DATABASE_NAME: ssb1_parquet
TABLE_ID: 114009
TABLE_NAME: dates
1 row inset (0.07 sec)

when you create / drop catalog , need not refresh catalog .

mysql>selectcount(*) frominternal.information_schema.metadata_name_ids\G; ***************************1. row ***************************count(*): 213011 row inset (0.34 sec)
mysql> drop catalog hive2;
Query OK, 0 rows affected (0.01 sec)
mysql>selectcount(*) frominternal.information_schema.metadata_name_ids\G; ***************************1. row ***************************count(*): 106651 row inset (0.04 sec) mysql> create catalog hive3 ... mysql>selectcount(*) frominternal.information_schema.metadata_name_ids\G; ***************************1. row ***************************count(*): 213011 row inset (0.32 sec)

create / drop table , need not refresh catalog .

mysql> CREATE TABLE IF NOT EXISTS demo.example_tbl ... ;
mysql>selectcount(*) frominternal.information_schema.metadata_name_ids\G; ***************************1. row ***************************count(*): 106661 row inset (0.04 sec)
mysql>droptabledemo.example_tbl;
Query OK, 0 rows affected (0.01 sec)
mysql>selectcount(*) frominternal.information_schema.metadata_name_ids\G; ***************************1. row ***************************count(*): 106651 row inset (0.04 sec)

you can set query time , prevent queries from taking too long .


fe.conf : query_metadata_name_ids_timeout the time used to obtain all tables in one database

add information_schema.profiling in order to Compatible with mysql

mysql>select*frominformation_schema.profiling;
Empty set (0.07 sec)
mysql>set profiling=1; Query OK, 0 rows affected (0.01 sec)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions · 2023-08-08T02:09:16Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-08-08T04:11:39Z

clang-tidy review says "All clean, LGTM! 👍"

morningman

please add regression test cases

morningman · 2023-08-09T15:56:12Z


+ //query internal.information_schema.simple_tables time
+ @VariableMgr.VarAttr(name = QUERY_SIMPLE_TABLES_TIMEOUT)
+ public int querySimpleTablesTimeout = 3;


I think it can be a FE config, not session variable.
Because we always set it globally

ok good idea

morningman · 2023-08-09T15:57:28Z

 SCH_COLUMN_STATISTICS,
- SCH_PARAMETERS;
+ SCH_PARAMETERS,
+	SCH_SIMPLE_TABLES,


Suggested change
SCH_SIMPLE_TABLES,
SCH_SIMPLE_TABLES,

morningman · 2023-08-09T15:57:38Z

- SCH_PARAMETERS;
+ SCH_PARAMETERS,
+	SCH_SIMPLE_TABLES,
+	SCH_PROFILING;


Suggested change
SCH_PROFILING;
SCH_PROFILING;

morningman · 2023-08-09T15:58:11Z

 }

+struct TSimpleTableStatus {
+	1: required string name


use 4 spaces instead of tab.
and use optional instead of required

ok good idea

morningman · 2023-08-09T15:58:16Z

+	2: required i64 id 
+}
+struct TListSimpleTableStatusResult {
+	1: required list<TSimpleTableStatus> tables 


morningman · 2023-08-09T16:00:10Z

+ .column("ROUTINE_TYPE", ScalarType.createVarchar(64))
+ .column("DATA_TYPEDTD_IDENDS", ScalarType.createVarchar(64))
+ .build()))
+ .put("simple_tables", new SchemaTable(SystemIdGenerator.getNextId(), "simple_tables", TableType.SCHEMA,


how about just name it as metadata_name_ids?

ok good idea

github-actions · 2023-08-11T05:53:27Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-08-11T08:06:51Z

clang-tidy review says "All clean, LGTM! 👍"

hubgeter · 2023-08-18T07:11:21Z

run buildall

github-actions · 2023-08-18T07:13:26Z

clang-tidy review says "All clean, LGTM! 👍"

hubgeter · 2023-08-18T11:31:57Z

run buildall

github-actions · 2023-08-18T11:37:51Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-08-18T12:27:53Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.7 seconds
stream load tsv: 535 seconds loaded 74807831229 Bytes, about 133 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17162394725 Bytes

morningman

LGTM

github-actions · 2023-08-20T14:26:21Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-08-20T14:26:23Z

PR approved by anyone and no changes requested.

github-actions · 2023-08-23T06:43:20Z

clang-tidy review says "All clean, LGTM! 👍"

morningman

LGTM

hubgeter · 2023-08-25T04:28:00Z

run buildall

github-actions · 2023-08-25T04:35:26Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-08-25T05:23:24Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.38 seconds
stream load tsv: 537 seconds loaded 74807831229 Bytes, about 132 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.5 seconds inserted 10000000 Rows, about 338K ops/s
storage size: 17162092058 Bytes

hubgeter · 2023-08-25T13:38:20Z

run buildall

github-actions · 2023-08-25T13:44:59Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-08-25T14:36:04Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.5 seconds
stream load tsv: 537 seconds loaded 74807831229 Bytes, about 132 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17162023308 Bytes

morningman

LGTM

github-actions · 2023-08-25T16:06:46Z

PR approved by at least one committer and no changes requested.

…tlogs,db,table and add profiling table in order to Compatible with mysql (apache#22702) add information_schema.metadata_name_idsfor quickly get catlogs,db,table. 1. table struct : ```mysql mysql> desc internal.information_schema.metadata_name_ids; +---------------+--------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +---------------+--------------+------+-------+---------+-------+ | CATALOG_ID | BIGINT | Yes | false | NULL | | | CATALOG_NAME | VARCHAR(512) | Yes | false | NULL | | | DATABASE_ID | BIGINT | Yes | false | NULL | | | DATABASE_NAME | VARCHAR(64) | Yes | false | NULL | | | TABLE_ID | BIGINT | Yes | false | NULL | | | TABLE_NAME | VARCHAR(64) | Yes | false | NULL | | +---------------+--------------+------+-------+---------+-------+ 6 rows in set (0.00 sec) mysql> select * from internal.information_schema.metadata_name_ids where CATALOG_NAME="hive1" limit 1 \G; *************************** 1. row *************************** CATALOG_ID: 113008 CATALOG_NAME: hive1 DATABASE_ID: 113042 DATABASE_NAME: ssb1_parquet TABLE_ID: 114009 TABLE_NAME: dates 1 row in set (0.07 sec) ``` 2. when you create / drop catalog , need not refresh catalog . ```mysql mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 21301 1 row in set (0.34 sec) mysql> drop catalog hive2; Query OK, 0 rows affected (0.01 sec) mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10665 1 row in set (0.04 sec) mysql> create catalog hive3 ... mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 21301 1 row in set (0.32 sec) ``` 3. create / drop table , need not refresh catalog . ```mysql mysql> CREATE TABLE IF NOT EXISTS demo.example_tbl ... ; mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10666 1 row in set (0.04 sec) mysql> drop table demo.example_tbl; Query OK, 0 rows affected (0.01 sec) mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10665 1 row in set (0.04 sec) ``` 4. you can set query time , prevent queries from taking too long . ``` fe.conf : query_metadata_name_ids_timeout the time used to obtain all tables in one database ``` 5. add information_schema.profiling in order to Compatible with mysql ```mysql mysql> select * from information_schema.profiling; Empty set (0.07 sec) mysql> set profiling=1; Query OK, 0 rows affected (0.01 sec) ```

…tlogs,db,table and add profiling table in order to Compatible with mysql (#22702) (#23753)

…y get catlogs,db,table and add profiling table in order to Compatible with mysql (#22702) (#23753)" This reverts commit 39c339c.

…y get catlogs,db,table and add profiling table in order to Compatible with mysql (#22702) (#23753)" (#23928) This reverts commit 39c339c.

…tlogs,db,table and add profiling table in order to Compatible with mysql (apache#22702) add information_schema.metadata_name_idsfor quickly get catlogs,db,table. 1. table struct : ```mysql mysql> desc internal.information_schema.metadata_name_ids; +---------------+--------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +---------------+--------------+------+-------+---------+-------+ | CATALOG_ID | BIGINT | Yes | false | NULL | | | CATALOG_NAME | VARCHAR(512) | Yes | false | NULL | | | DATABASE_ID | BIGINT | Yes | false | NULL | | | DATABASE_NAME | VARCHAR(64) | Yes | false | NULL | | | TABLE_ID | BIGINT | Yes | false | NULL | | | TABLE_NAME | VARCHAR(64) | Yes | false | NULL | | +---------------+--------------+------+-------+---------+-------+ 6 rows in set (0.00 sec) mysql> select * from internal.information_schema.metadata_name_ids where CATALOG_NAME="hive1" limit 1 \G; *************************** 1. row *************************** CATALOG_ID: 113008 CATALOG_NAME: hive1 DATABASE_ID: 113042 DATABASE_NAME: ssb1_parquet TABLE_ID: 114009 TABLE_NAME: dates 1 row in set (0.07 sec) ``` 2. when you create / drop catalog , need not refresh catalog . ```mysql mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 21301 1 row in set (0.34 sec) mysql> drop catalog hive2; Query OK, 0 rows affected (0.01 sec) mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10665 1 row in set (0.04 sec) mysql> create catalog hive3 ... mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 21301 1 row in set (0.32 sec) ``` 3. create / drop table , need not refresh catalog . ```mysql mysql> CREATE TABLE IF NOT EXISTS demo.example_tbl ... ; mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10666 1 row in set (0.04 sec) mysql> drop table demo.example_tbl; Query OK, 0 rows affected (0.01 sec) mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10665 1 row in set (0.04 sec) ``` 4. you can set query time , prevent queries from taking too long . ``` fe.conf : query_metadata_name_ids_timeout the time used to obtain all tables in one database ``` 5. add information_schema.profiling in order to Compatible with mysql ```mysql mysql> select * from information_schema.profiling; Empty set (0.07 sec) mysql> set profiling=1; Query OK, 0 rows affected (0.01 sec) ```

…tlogs,db,table and add profiling table in order to Compatible with mysql (#22702) (#23980)

…y get catlogs,db,table and add profiling table in order to Compatible with mysql (#22702) (#23980)" This reverts commit 01ee595.

…y get catlogs,db,table and add profiling table in order to Compatible with mysql (#22702) (#23980)" (#24002) This reverts commit 01ee595.

…tlogs,db,table and add profiling table in order to Compatible with mysql (apache#22702) add information_schema.metadata_name_idsfor quickly get catlogs,db,table. 1. table struct : ```mysql mysql> desc internal.information_schema.metadata_name_ids; +---------------+--------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +---------------+--------------+------+-------+---------+-------+ | CATALOG_ID | BIGINT | Yes | false | NULL | | | CATALOG_NAME | VARCHAR(512) | Yes | false | NULL | | | DATABASE_ID | BIGINT | Yes | false | NULL | | | DATABASE_NAME | VARCHAR(64) | Yes | false | NULL | | | TABLE_ID | BIGINT | Yes | false | NULL | | | TABLE_NAME | VARCHAR(64) | Yes | false | NULL | | +---------------+--------------+------+-------+---------+-------+ 6 rows in set (0.00 sec) mysql> select * from internal.information_schema.metadata_name_ids where CATALOG_NAME="hive1" limit 1 \G; *************************** 1. row *************************** CATALOG_ID: 113008 CATALOG_NAME: hive1 DATABASE_ID: 113042 DATABASE_NAME: ssb1_parquet TABLE_ID: 114009 TABLE_NAME: dates 1 row in set (0.07 sec) ``` 2. when you create / drop catalog , need not refresh catalog . ```mysql mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 21301 1 row in set (0.34 sec) mysql> drop catalog hive2; Query OK, 0 rows affected (0.01 sec) mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10665 1 row in set (0.04 sec) mysql> create catalog hive3 ... mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 21301 1 row in set (0.32 sec) ``` 3. create / drop table , need not refresh catalog . ```mysql mysql> CREATE TABLE IF NOT EXISTS demo.example_tbl ... ; mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10666 1 row in set (0.04 sec) mysql> drop table demo.example_tbl; Query OK, 0 rows affected (0.01 sec) mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10665 1 row in set (0.04 sec) ``` 4. you can set query time , prevent queries from taking too long . ``` fe.conf : query_metadata_name_ids_timeout the time used to obtain all tables in one database ``` 5. add information_schema.profiling in order to Compatible with mysql ```mysql mysql> select * from information_schema.profiling; Empty set (0.07 sec) mysql> set profiling=1; Query OK, 0 rows affected (0.01 sec) ```

hubgeter changed the title ~~add information_schema.simple_tables for quickly get catlogs,db,table.~~[feature](information_schema)add simple_tables for quickly get catlogs,db,table and add profiling table in order to Compatible with mysqlAug 8, 2023

morningman added the dev/2.0.1 label Aug 9, 2023

morningman reviewed Aug 9, 2023
View reviewed changes

hubgeter force-pushed the tables branch from afa864b to 9a5f1dfCompare August 11, 2023 05:44

hubgeter force-pushed the tables branch from 1094ef3 to 37c6cbeCompare August 18, 2023 07:06

morningman previously approved these changes Aug 20, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 20, 2023

github-actionsBot added the reviewed label Aug 20, 2023

hubgeter dismissed morningman’s stale review via 1022c1eAugust 23, 2023 06:35

hubgeter force-pushed the tables branch from efe4ce8 to 1022c1eCompare August 23, 2023 06:35

morningman previously approved these changes Aug 24, 2023
View reviewed changes

hubgeter dismissed morningman’s stale review via b97a572August 25, 2023 04:27

hubgeter force-pushed the tables branch from 1022c1e to b97a572Compare August 25, 2023 04:27

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Aug 25, 2023

fix regression test
b450f3c

hubgeter force-pushed the tables branch from b97a572 to b450f3cCompare August 25, 2023 13:36

morningman approved these changes Aug 25, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 25, 2023

xiaokang added dev/2.0.2 and removed dev/2.0.1 labels Aug 30, 2023

BePPPower approved these changes Aug 31, 2023
View reviewed changes

morningman merged commit e680d42 into apache:masterAug 31, 2023

xiaokang added the merge_conflict label Aug 31, 2023

xiaokang pushed a commit that referenced this pull request Sep 5, 2023
[feature](information_schema)add metadata_name_ids for quickly get ca…
39c339c
…tlogs,db,table and add profiling table in order to Compatible with mysql (#22702) (#23753)

BiteTheDDDDt added a commit that referenced this pull request Sep 5, 2023
Revert "[feature](information_schema)add metadata_name_ids for quickl…
e6e33d7
…y get catlogs,db,table and add profiling table in order to Compatible with mysql (#22702) (#23753)" This reverts commit 39c339c.

BiteTheDDDDt mentioned this pull request Sep 5, 2023
[Chore](revert) revert 23753 on 2.0 #23928
Merged

hubgeter mentioned this pull request Sep 6, 2023
[feature](information_schema)add metadata_name_ids for quickly get c… #23980
Merged

xiaokang pushed a commit that referenced this pull request Sep 6, 2023
[feature](information_schema)add metadata_name_ids for quickly get ca…
01ee595
…tlogs,db,table and add profiling table in order to Compatible with mysql (#22702) (#23980)

xiaokang added a commit that referenced this pull request Sep 6, 2023
Revert "[feature](information_schema)add metadata_name_ids for quickl…
e8744ff
…y get catlogs,db,table and add profiling table in order to Compatible with mysql (#22702) (#23980)" This reverts commit 01ee595.

xiaokang added a commit that referenced this pull request Sep 6, 2023
Revert "[feature](information_schema)add metadata_name_ids for quickl…
7cc1d65
…y get catlogs,db,table and add profiling table in order to Compatible with mysql (#22702) (#23980)" (#24002) This reverts commit 01ee595.

morningman mentioned this pull request Sep 8, 2023
[feature](information_schema)add metadata_name_ids for quickly get catlogs,db,table and add profiling table in order to Compatible with mysql #24059
Merged

morningman added dev/2.0.2-merged and removed dev/2.0.2 labels Sep 8, 2023

xiaokang mentioned this pull request Sep 30, 2023
Release Note 2.0.2 #25011
Closed

xiaokang mentioned this pull request Dec 3, 2023
Release Note 2.0.3 #27909
Closed

Uh oh!

Conversation

hubgeter commented Aug 8, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

github-actionsBot commented Aug 8, 2023

Uh oh!

github-actionsBot commented Aug 8, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Aug 11, 2023

Uh oh!

github-actionsBot commented Aug 11, 2023

Uh oh!

hubgeter commented Aug 18, 2023

Uh oh!

github-actionsBot commented Aug 18, 2023

Uh oh!

hubgeter commented Aug 18, 2023

Uh oh!

github-actionsBot commented Aug 18, 2023

Uh oh!

hello-stephen commented Aug 18, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Aug 20, 2023

Uh oh!

github-actionsBot commented Aug 20, 2023

Uh oh!

github-actionsBot commented Aug 23, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

hubgeter commented Aug 25, 2023

Uh oh!

github-actionsBot commented Aug 25, 2023

Uh oh!

hello-stephen commented Aug 25, 2023

Uh oh!

hubgeter commented Aug 25, 2023

Uh oh!

github-actionsBot commented Aug 25, 2023

Uh oh!

hello-stephen commented Aug 25, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Aug 25, 2023

Uh oh!

Reviewers

Assignees

Labels

hubgeter commented Aug 8, 2023•
edited
Loading