[Doris On ES][Feature] Support create table with wildcard or aliase index

[wuyunfeng]

#3967

Create multiple index with same mapping in ES:

PUT wildcard_1
{
"settings": {
"number_of_replicas": 0,
"number_of_shards": 1
},
"mappings": {
"doc": {
"properties": {
"k1": {
"type": "integer"
},
"k2": {
"type": "keyword"
}
}
}
}
}
PUT wildcard_2
{
"settings": {
"number_of_replicas": 0,
"number_of_shards": 1
},
"mappings": {
"doc": {
"properties": {
"k1": {
"type": "integer"
},
"index": {
"type": "keyword"
}
}
}
}
}
PUT wildcard_3
{
"settings": {
"number_of_replicas": 0,
"number_of_shards": 1
},
"mappings": {
"doc": {
"properties": {
"k1": {
"type": "integer"
},
"index": {
"type": "keyword"
}
}
}
}
}

k2 used for indicating the data from which index

Create wildcard_alias aliases contains those above three index:

POST /_aliases
{
"actions": [
{
"add": {
"indices": [
"wildcard_3",
"wildcard_2",
"wildcard_1"
],
"alias": "wildcard_alias"
}
}
]
}

Create ES external table within Doris:

• Support aliase matched index:

CREATE EXTERNAL TABLE `wildcard_alias` (
`k1` int COMMENT "",
`k2` varchar COMMENT ""
) ENGINE=ELASTICSEARCH
PROPERTIES (
"hosts" = "http://127.0.0.1:8200",
"user" = "root",
"password" = "root",
"index" = "wildcard_alias",
"type" = "doc",
"version" = "6.5.3",
"enable_docvalue_scan" = "true"
);

• Support wildcard matched index (pattern: wildcard_*):

CREATE EXTERNAL TABLE `wildcard_alias` (
`k1` int COMMENT "",
`k2` varchar COMMENT ""
) ENGINE=ELASTICSEARCH
PROPERTIES (
"hosts" = "http://10.74.167.16:8200",
"user" = "root",
"password" = "root",
"index" = "wildcard_*",
"type" = "doc",
"version" = "6.5.3",
"enable_docvalue_scan" = "true"
);

And execute SQL on Doris:

 select * from wildcard_alias;
or
select * from wildcard;

Finally we get all document within those three index:

+------+-------------+
| k1 | k2 |
+------+-------------+
| 111 | wiladcard_1 |
| 1111 | wiladcard_1 |
| 1112 | wiladcard_1 |
| 110 | wiladcard_1 |
| 222 | wiladcard_2 |
| 223 | wiladcard_2 |
| 223 | wiladcard_2 |
| 33 | wiladcard_3 |
| 33 | wiladcard_3 |
| 33 | wiladcard_3 |
| 33 | wiladcard_3 |
| 33 | wiladcard_3 |
+------+-------------+

BTW: predicates which can push down would push down to all related indices

[BabySid]

looks good.
but i have a question: sql like 'select k1 from wildcard_alias where k1 > '2020-05-20'', DSL will run on all related indices? or just on wildcard_alias_2020_05_21, wildcard_alias_2020_05_22...

[wuyunfeng]

@blackfox1983 yes, run on all related indices. process predicate is independent with processing resolved indidces

[morningman]

LGTM