[Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable by hubgeter · Pull Request #41659 · apache/doris

hubgeter · 2024-10-10T08:33:26Z

Proposed changes

Because ExternalTable will initialize the previously uninitialized table when getCachedRowCount(), which is unnecessary. So for the uninitialized table, we directly return -1.
This will increase the speed of our query information_schema.tables.

doris-robot · 2024-10-10T08:33:31Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

hubgeter · 2024-10-10T08:33:37Z

run buildall

hubgeter · 2024-10-10T10:59:51Z

run buildall

hubgeter · 2024-10-10T14:18:26Z

run buildall

morningman · 2024-10-11T04:01:50Z

+ // ExternalTable.getRowCount(), but this is not very meaningful and time-consuming.
+ // The getCachedRowCount() function is only used when `show table` and querying `information_schema.tables`.
+ if (!checkInitialized()) {
+ return -2;


i will fix it

morningman · 2024-10-11T04:02:16Z

 return false;
 }

+ public boolean checkInitialized() {


No need this method, we can use check objectCreated

i will remove this method.

morningman · 2024-10-11T04:04:18Z

 private boolean enableHmsEventsIncrementalSync = false;

+ //for "type" = "hms" , but is iceberg table.
+ HiveCatalog icebergHiveCatalog;


Suggested change
HiveCatalogicebergHiveCatalog;
privateHiveCatalogicebergHiveCatalog;

i will fix it

morningman · 2024-10-11T04:06:39Z

 }

+ public Catalog getIcebergHiveCatalog() {
+ if (icebergHiveCatalog == null) {


This method may be called concurrently

for prevent this, i will move icebergHiveCatalog init to initLocalObjectsImpl()

morningman · 2024-10-11T04:14:00Z

- HiveCatalog hiveCatalog = new HiveCatalog();
- hiveCatalog.setConf(conf);
-
- if (props.containsKey(HMSExternalCatalog.BIND_BROKER_NAME)) {


This part is missing?

I kept this part and removed the else part. As it seemed to me that the two were duplicates. The else part just initialize hivecatalog use uriproperty.

…ount when reading ExternalTable

hubgeter · 2024-10-11T06:42:15Z

run buildall

morningman

LGTM

github-actions · 2024-10-11T10:18:41Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-10-11T10:18:43Z

PR approved by anyone and no changes requested.

wuwenchi · 2024-10-11T10:33:51Z

+ HiveCatalog hiveCatalog = new org.apache.iceberg.hive.HiveCatalog();
+ hiveCatalog.setConf(externalCatalog.getConfiguration());
+
+ Map<String, String> catalogProperties = externalCatalog.getProperties();
+ String metastoreUris = catalogProperties.getOrDefault(HMSProperties.HIVE_METASTORE_URIS, "");
+ catalogProperties.put(CatalogProperties.URI, metastoreUris);
+
+ hiveCatalog.initialize(name, catalogProperties);
+ return hiveCatalog;


Is same as IcebergHMSExternalCatalog.initCatalog ?

yes . so now IcebergHMSExternalCatalog.initCatalog() method use this method.

you can ref : https://github.com/apache/doris/pull/41659/files#diff-e6e9c0730301ce99354da7a68c322f1c94d656033b25ab525eab6f17eb9dce7dL41

…ount when reading ExternalTable (apache#41659) ## Proposed changes Because ExternalTable will initialize the previously uninitialized table when `getCachedRowCount()`, which is unnecessary. So for the uninitialized table, we directly return -1. This will increase the speed of our query `information_schema.tables`.

…ount when reading ExternalTable (apache#41659) Because ExternalTable will initialize the previously uninitialized table when `getCachedRowCount()`, which is unnecessary. So for the uninitialized table, we directly return -1. This will increase the speed of our query `information_schema.tables`.

…ount when reading ExternalTable (apache#41659) ## Proposed changes Because ExternalTable will initialize the previously uninitialized table when `getCachedRowCount()`, which is unnecessary. So for the uninitialized table, we directly return -1. This will increase the speed of our query `information_schema.tables`.

…ount when reading ExternalTable (#41659) (#41959) bp #41659 ## Proposed changes Because ExternalTable will initialize the previously uninitialized table when `getCachedRowCount()`, which is unnecessary. So for the uninitialized table, we directly return -1. This will increase the speed of our query `information_schema.tables`.

…ount when reading ExternalTable (#41659) (#41962) bp #41659 ## Proposed changes Because ExternalTable will initialize the previously uninitialized table when `getCachedRowCount()`, which is unnecessary. So for the uninitialized table, we directly return -1. This will increase the speed of our query `information_schema.tables`.

…ount when reading ExternalTable (apache#41659) ## Proposed changes Because ExternalTable will initialize the previously uninitialized table when `getCachedRowCount()`, which is unnecessary. So for the uninitialized table, we directly return -1. This will increase the speed of our query `information_schema.tables`.

hubgeter force-pushed the opt_infomation_schema_tables branch from 90d6893 to c3535c9Compare October 10, 2024 14:18

morningman added dev/2.1.x dev/3.0.x labels Oct 11, 2024

morningman reviewed Oct 11, 2024
View reviewed changes

hubgeter force-pushed the opt_infomation_schema_tables branch from c3535c9 to 3d68a83Compare October 11, 2024 06:40

hubgeter added 3 commits October 11, 2024 14:42

[Enhancement](ExternalTable)Optimize the performance of getCachedRowC…
089629d
…ount when reading ExternalTable

fix case
d92304c

fix comment
7526b02

hubgeter force-pushed the opt_infomation_schema_tables branch from 3d68a83 to 7526b02Compare October 11, 2024 06:42

morningman approved these changes Oct 11, 2024
View reviewed changes

github-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Oct 11, 2024

wuwenchi reviewed Oct 11, 2024
View reviewed changes

wuwenchi approved these changes Oct 11, 2024
View reviewed changes

morningman merged commit 127ac8d into apache:masterOct 13, 2024

hubgeter mentioned this pull request Oct 16, 2024
[Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable (#41659) #41959
Merged

hubgeter mentioned this pull request Oct 16, 2024
[Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable (#41659) #41962
Merged

yiguolei added dev/2.1.7-merged and removed dev/2.1.x labels Oct 17, 2024

morningman added dev/3.0.3-merged and removed dev/3.0.x labels Oct 18, 2024

gavinchou mentioned this pull request Nov 26, 2024
Release Note 3.0.3 #44522
Closed

	HiveCatalogicebergHiveCatalog;
	privateHiveCatalogicebergHiveCatalog;

Uh oh!

Conversation

hubgeter commented Oct 10, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented Oct 10, 2024

Uh oh!

hubgeter commented Oct 10, 2024

Uh oh!

hubgeter commented Oct 10, 2024

Uh oh!

hubgeter commented Oct 10, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hubgeterOct 11, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hubgeter commented Oct 11, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Oct 11, 2024

Uh oh!

github-actionsBot commented Oct 11, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

hubgeter commented Oct 10, 2024•
edited
Loading

hubgeterOct 11, 2024•
edited
Loading