[fix](ES Catalog)Only like on keyword can be applied to wildcard query by qidaye · Pull Request #41176 · apache/doris

qidaye · 2024-09-23T14:11:03Z

Proposed changes

We map text and keyword both to string type in Doris. When enable like_push_down, we translate like to wildcard query in ES, which will lead unexpected result in text field. We should stick to keyword with wildcard query.

Add column2typeMap in EsTable to save the mapping of column_name to ES field data type.
Add new class EsSchemaCacheValue to get schema and column to type map
Init column2typeMap when cache init and build query process of ES external table
Support LIKE functionCallExpr for Nereids planner.
Add end to end like predicate test cases and UTs

doris-robot · 2024-09-23T14:11:09Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

qidaye · 2024-09-23T14:11:11Z

run buildall

xiaokang

Do you have a end to end testcase for text_field LIKE xxx ?

qidaye · 2024-09-25T13:50:28Z

run buildall

xiaokang

LGTM

github-actions · 2024-09-25T14:36:14Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-09-25T14:36:16Z

PR approved by anyone and no changes requested.

qidaye · 2024-09-26T08:16:00Z

run buildall

qidaye · 2024-10-08T02:47:08Z

run buildall

We map `text` and `keyword` both to `string` type in Doris. When enable `like_push_down`, we translate like to wildcard query in ES, which will lead unexpected result in `text` field. We should stick to `keyword` with wildcard query. 1. Add `column2typeMap` in `EsTable` to save the mapping of column_name to ES field data type. 2. Add new class `EsSchemaCacheValue` to get schema and column to type map 2. Init `column2typeMap` when cache init and build query process of ES external table

2. Add end to end test case for like.

qidaye · 2024-11-05T07:18:29Z

run buildall

morningman

LGTM

github-actions · 2024-11-06T15:43:41Z

PR approved by at least one committer and no changes requested.

airborne12

LGTM

#41176) ## Proposed changes We map `text` and `keyword` both to `string` type in Doris. When enable `like_push_down`, we translate like to wildcard query in ES, which will lead unexpected result in `text` field. We should stick to `keyword` with wildcard query. 1. Add `column2typeMap` in `EsTable` to save the mapping of column_name to ES field data type. 2. Add new class `EsSchemaCacheValue` to get schema and column to type map 3. Init `column2typeMap` when cache init and build query process of ES external table 4. Support LIKE functionCallExpr for Nereids planner. 5. Add end to end like predicate test cases and UTs

apache#41176) We map `text` and `keyword` both to `string` type in Doris. When enable `like_push_down`, we translate like to wildcard query in ES, which will lead unexpected result in `text` field. We should stick to `keyword` with wildcard query. 1. Add `column2typeMap` in `EsTable` to save the mapping of column_name to ES field data type. 2. Add new class `EsSchemaCacheValue` to get schema and column to type map 3. Init `column2typeMap` when cache init and build query process of ES external table 4. Support LIKE functionCallExpr for Nereids planner. 5. Add end to end like predicate test cases and UTs

…ildcard query (#43400) Cherry-picked from #41176 Co-authored-by: qiye <jianliang5669@gmail.com>

#41176) (#43459) bp #41176

…ildcard query (#43399) Cherry-picked from #41176 Co-authored-by: qiye <jianliang5669@gmail.com> Co-authored-by: airborne12 <airborne08@gmail.com>

apache#41176) ## Proposed changes We map `text` and `keyword` both to `string` type in Doris. When enable `like_push_down`, we translate like to wildcard query in ES, which will lead unexpected result in `text` field. We should stick to `keyword` with wildcard query. 1. Add `column2typeMap` in `EsTable` to save the mapping of column_name to ES field data type. 2. Add new class `EsSchemaCacheValue` to get schema and column to type map 3. Init `column2typeMap` when cache init and build query process of ES external table 4. Support LIKE functionCallExpr for Nereids planner. 5. Add end to end like predicate test cases and UTs

xiaokang reviewed Sep 24, 2024
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/datasource/es/EsUtil.java

qidaye added dev/2.0.x dev/2.1.x labels Sep 25, 2024

xiaokang previously approved these changes Sep 25, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 25, 2024

github-actionsBot added the reviewed label Sep 25, 2024

qidaye dismissed xiaokang’s stale review via b20d14dSeptember 26, 2024 08:14

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Sep 26, 2024

qidaye force-pushed the fix_es_catalog_like branch from b20d14d to 97e5c2fCompare October 8, 2024 02:46

morningman added the dev/3.0.x label Oct 14, 2024

qidaye added 4 commits November 5, 2024 14:44

1. support like functionCallExpr for Nereids
1f7a722
2. Add end to end test case for like.

restore default type to STRING
f74a26b

fix tests
e7dc509

qidaye force-pushed the fix_es_catalog_like branch from 97e5c2f to e7dc509Compare November 5, 2024 07:18

morningman approved these changes Nov 6, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Nov 6, 2024

airborne12 approved these changes Nov 7, 2024
View reviewed changes

airborne12 merged commit bb5533d into apache:masterNov 7, 2024

github-actionsBot mentioned this pull request Nov 7, 2024
branch-3.0: [fix](ES Catalog)Only like on keyword can be applied to wildcard query #43399
Merged

github-actionsBot mentioned this pull request Nov 7, 2024
branch-2.1: [fix](ES Catalog)Only like on keyword can be applied to wildcard query #43400
Merged

qidaye deleted the fix_es_catalog_like branch November 7, 2024 09:10

qidaye mentioned this pull request Nov 7, 2024
[fix](ES Catalog)Only like on keyword can be applied to wildcard query (#41176) #43459
Merged

airborne12 pushed a commit that referenced this pull request Nov 7, 2024
branch-2.1: [fix](ES Catalog)Only like on keyword can be applied to w…
7de7541
…ildcard query (#43400) Cherry-picked from #41176 Co-authored-by: qiye <jianliang5669@gmail.com>

wm1581066 added dev/2.1.8-merged and removed dev/2.1.x labels Nov 7, 2024

airborne12 pushed a commit that referenced this pull request Nov 8, 2024
[fix](ES Catalog)Only like on keyword can be applied to wildcard query (
e6b5108
#41176) (#43459) bp #41176

airborne12 added dev/2.0.16-merged-deprecated 2.0.16 is NOT and will NOT be released and removed dev/2.0.x labels Nov 8, 2024

qidaye added dev/3.0.3-merged dev/3.0.x and removed dev/3.0.x dev/3.0.3-merged labels Nov 8, 2024

airborne12 added dev/3.0.3-merged and removed dev/3.0.x labels Nov 8, 2024

Uh oh!

Conversation

qidaye commented Sep 23, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented Sep 23, 2024

Uh oh!

qidaye commented Sep 23, 2024

Uh oh!

xiaokang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

qidaye commented Sep 25, 2024

Uh oh!

xiaokang left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 25, 2024

Uh oh!

github-actionsBot commented Sep 25, 2024

Uh oh!

qidaye commented Sep 26, 2024

Uh oh!

qidaye commented Oct 8, 2024

Uh oh!

qidaye commented Nov 5, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Nov 6, 2024

Uh oh!

airborne12 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

qidaye commented Sep 23, 2024•
edited
Loading