[Doris On ES] Split /_cluster/state to [indexName/_mappings, indexName/_search_shards] by stalary · Pull Request #3454 · apache/doris

stalary · 2020-05-02T16:15:41Z

Split /_cluster/state into /_mapping and /_search_shards requests to reduce permissions and make the logic clearer
Rename part es related objects to make their representation more accurate
Simply support docValue and Fields in alias mode, and take the first one by default

#3311

…_shards]

imay · 2020-05-05T07:05:35Z

@wuyunfeng
Could you please review this PR?

@stalary
You'd better add more information about this change list in commit message

wuyunfeng · 2020-05-05T13:44:46Z

OK

…to stalary_enhancement

wuyunfeng

Thanks @stalary , I left some comments.

…to stalary_enhancement

# Conflicts: # fe/src/main/java/org/apache/doris/external/elasticsearch/EsRestClient.java # fe/src/main/java/org/apache/doris/external/elasticsearch/EsStateStore.java # fe/src/test/java/org/apache/doris/es/EsStateStoreTest.java

…to stalary_enhancement # Conflicts: # fe/src/main/java/org/apache/doris/external/elasticsearch/EsRestClient.java # fe/src/main/java/org/apache/doris/external/elasticsearch/EsStateStore.java # fe/src/test/java/org/apache/doris/es/EsStateStoreTest.java

BabySid

加下必要的注释提高review的效率和质量

morningman

LGTM

morningman

LGTM

…e/_search_shards] (apache#3454) 1. Split /_cluster/state into /_mapping and /_search_shards requests to reduce permissions and make the logic clearer 2. Rename part es related objects to make their representation more accurate 3. Simply support docValue and Fields in alias mode, and take the first one by default apache#3311

…ade FE (#3961) ISSUE:#3960 PR #3454 introduce the caching for EsClient, but the initialization of the client was only during editlog replay, all this work should done also during image replay. This happens when restart or upgrade FE BTW: modify a UT failure for metric

…ade FE (apache#3961) ISSUE:apache#3960 PR apache#3454 introduce the caching for EsClient, but the initialization of the client was only during editlog replay, all this work should done also during image replay. This happens when restart or upgrade FE BTW: modify a UT failure for metric

…4012) After PR #3454 was merged, we should refactor and reorganize some logic for long-term sustainable iteration for Doris On ES. To facilitate code review，I would divided into this work to multiple PRs (some other WIP work I also need to think carefully) This PR include: 1. introduce SearchContext for all state we needed 2. divide meta-sync logic into three phase 3. modify some logic processing 4. introduce version detect logic for future using

…pache#4012) After PR apache#3454 was merged, we should refactor and reorganize some logic for long-term sustainable iteration for Doris On ES. To facilitate code review，I would divided into this work to multiple PRs (some other WIP work I also need to think carefully) This PR include: 1. introduce SearchContext for all state we needed 2. divide meta-sync logic into three phase 3. modify some logic processing 4. introduce version detect logic for future using

stalary added 3 commits May 2, 2020 11:31

MOD: fix varchar max error
e2d09f5

MOD: Split /_cluster/state to [indexName/_mappings, indexName/_search…
b6ff6b2
…_shards]

MOD: Mod some details
d858a74

imay self-assigned this May 5, 2020

BabySid reviewed May 6, 2020
View reviewed changes

Comment threaddocs/en/sql-reference/sql-statements/Data Types/VARCHAR.md Outdated

BabySid reviewed May 6, 2020
View reviewed changes

Comment threadfe/src/main/java/org/apache/doris/external/EsIndexState.java
Comment threadfe/src/main/java/org/apache/doris/external/EsIndexState.java Outdated

MOD: Cancel the folding line，Remove the changes to varchar.md
a3d2928

wuyunfeng requested changes May 7, 2020
View reviewed changes

stalary added 2 commits May 18, 2020 22:28

Merge branch 'master' ofhttps://github.com/apache/incubator-dorisin…
558b8ee
…to stalary_enhancement

MOD: Make some changes based on comments
d25b21f

wuyunfeng requested changes May 19, 2020
View reviewed changes

Comment threadfe/src/main/java/org/apache/doris/external/EsRestClient.java Outdated
Comment threadfe/src/main/java/org/apache/doris/external/EsStateStore.java Outdated
Comment threadfe/src/main/java/org/apache/doris/external/EsStateStore.java Outdated

stalary added 6 commits May 24, 2020 12:38

MOD: make some changes based on the comments
407e913

FIX: fix dovValueField bug
f06f846

MOD: remove useless import
a381b68

MOD
8400246

Merge branch 'master' ofhttps://github.com/apache/incubator-dorisin…
7b79445
…to stalary_enhancement

MOD: Supplement Doris on es documentation
451b9a6

wuyunfeng reviewed May 26, 2020
View reviewed changes

Comment threadfe/src/main/java/org/apache/doris/external/EsRestClient.java Outdated

wuyunfeng requested changes May 26, 2020
View reviewed changes

stalary added 5 commits June 10, 2020 17:58

MOD: refactor code
3e768c8

Merge branch 'master' into stalary_enhancement
2ed77b6
# Conflicts: # fe/src/main/java/org/apache/doris/external/elasticsearch/EsRestClient.java # fe/src/main/java/org/apache/doris/external/elasticsearch/EsStateStore.java # fe/src/test/java/org/apache/doris/es/EsStateStoreTest.java

MOD: Adjust import order, resolve conflicts
b975c7a

MOD: adjust import order
3782f4b

wuyunfeng requested changes Jun 12, 2020
View reviewed changes

Comment threadfe/src/main/java/org/apache/doris/external/elasticsearch/EsFieldInfo.java Outdated
Comment threadfe/src/main/java/org/apache/doris/external/elasticsearch/EsFieldInfo.java Outdated
Comment threadfe/src/main/java/org/apache/doris/external/elasticsearch/EsRestClient.java Outdated

wuyunfeng requested changes Jun 12, 2020
View reviewed changes

MOD: Abstract code
97b2338

wuyunfeng requested changes Jun 16, 2020
View reviewed changes

BabySid reviewed Jun 19, 2020
View reviewed changes

wuyunfeng requested changes Jun 19, 2020
View reviewed changes

Comment threadfe/src/main/java/org/apache/doris/external/elasticsearch/EsRestClient.java Outdated
Comment threadfe/src/main/java/org/apache/doris/external/elasticsearch/EsShardPartitions.java
Comment threadfe/src/main/java/org/apache/doris/external/elasticsearch/EsStateStore.java Outdated

MOD: Refactor some places
95b5117

wuyunfeng requested changes Jun 22, 2020
View reviewed changes

Comment threadfe/src/main/java/org/apache/doris/external/elasticsearch/EsRepository.java Outdated
Comment threadfe/src/main/java/org/apache/doris/external/elasticsearch/EsRepository.java Outdated

MOD: EsRepository logic moves to EsTable
5e42b5b

wuyunfeng requested changes Jun 23, 2020
View reviewed changes

Comment threadfe/src/main/java/org/apache/doris/external/elasticsearch/EsRepository.java Outdated
Comment threadfe/src/main/java/org/apache/doris/external/elasticsearch/EsRepository.java Outdated

stalary added 4 commits June 23, 2020 13:58

Merge branch 'master' into stalary_enhancement
e3748ec

MOD: Abstract code, change the function name
a7ba9d1

MOD: format
303d0a2

MOD: mod error info
15ba2d3

morningman previously approved these changes Jun 23, 2020
View reviewed changes

morningman added approved Indicates a PR has been approved by one committer. area/doris-on-es Issues or PRs related to Doris on ElasticSearch kind/improvement labels Jun 23, 2020

wuyunfeng previously approved these changes Jun 23, 2020
View reviewed changes

wutiangan reviewed Jun 24, 2020
View reviewed changes

Comment threadfe/src/main/java/org/apache/doris/catalog/EsTable.java Outdated

MOD: format code
5fbae71

stalary dismissed stale reviews from wuyunfeng and morningman via 5fbae71June 24, 2020 08:57

morningman approved these changes Jun 26, 2020
View reviewed changes

morningman merged commit a894b1e into apache:masterJun 26, 2020

This was referenced Jun 28, 2020
[Doris On ES][Bug-fix] Sync ES metadata failure after restart or upgrade FE #3961
Merged
[Doris On ES] Support ES alias and wildcard index #3967
Closed

wuyunfeng mentioned this pull request Jul 3, 2020
[Doris On ES][Refactor] refactor and enchanment ES sync meta logic #4012
Merged

stalary deleted the stalary_enhancement branch July 19, 2020 08:05

Uh oh!

Conversation

stalary commented May 2, 2020• edited by morningman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

imay commented May 5, 2020

Uh oh!

wuyunfeng commented May 5, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wuyunfeng left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BabySid left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

stalary commented May 2, 2020•
edited by morningman
Loading