[feature](cold-hot) support s3 resource

[qidaye]

Proposed changes

issue：#8807

1. Support create s3 resource
2. Support alter resource
3. Add check whether the resource is in use before deleting it
4. Support create OLAP table with s3 resource
5. Add remote_storage_resource and remote_storage_cooldown_time in DataProperty.
6. Support for modifying a single data property, do not support modify remote_storage_resource

Problem Summary:

Describe the overview of changes.

Checklist(Required)

1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

[pengxiangyu]

LGTM

[github-actions]

PR approved by anyone and no changes requested.

[yiguolei]

Could you please provide the sql references? For example how to define S3 resource and how to use it to create table

[yiguolei]

just use storage_resource= xxx , if it is defined then it implies that the cold storage medium type is s3 or hdd.

[qidaye]

remote_storage_resource must be used with storage_cold_medium. Standalone use is not supported.
There is a check in PropertyAnalyzer.

[yiguolei]

I don't think we need the conception of remote_storage_resource, if there is remote_storage_resource, there maybe a local_storage_resource.
And we could create a table on S3 or HDFS directly not in cold and hot scenario but in decouple compute and storage scenario.

[yiguolei]

cold_storage_resource = "xxx,xxxx,xxxx" it should support a array, because S3 bucket's bandwidth is too small, maybe we need multiple bucket to extend the bandwidth.

[qidaye]

Use the same resource when creating a table to facilitate data management.
Since the remote_storage_resource is partition level, user can change the resource when creating a new partition or modify by ALTER TABEL clause. In this way, a table can use multiple buckets.

[yiguolei]

If user could specify multi resources then he do not care the bandwidth in the future. If there is only one resource he need to monitor and optimize the resource usage by using alter table clause. It is too hard for the user.

[qidaye]

Could you please provide the sql references? For example how to define S3 resource and how to use it to create table

You can find the syntax in issue #8807

[yiguolei]

Do not need storage cold medium, If user specified storage resource then it implies the cold storage is S3 or HDFS.

[qidaye]

There is also a old logic data migration from SSD to HDD, we use both storage_cold_medium and remote_storage_resource to distinguish them

[yiguolei]

yeah

[yiguolei]

If BE have ssd,hdd,and s3 resource, how to define the operation from ssd->hdd->s3?

[yiguolei]

Use gson to do serialize and deserialize

[qidaye]

DataProperty is a already exist class. I can not change the serialize/deserialize to GSON since the backward compatible

[yiguolei]

....
for read method, you could use
if (meta version > 108)
{
gson.readxxx
}
else
{
read field
read filed
...
}

for write method, you could use
if (meta version > 108)
{
gson.writexxx
}
else
{
write field
write field
....
}

[yiguolei]

I think we should use system table instead of show command....
For example, select * from information_schema.resources where xxxx;

[qidaye]

This is a good suggestion, this pr is mainly for adding s3 resource, using the historical resource design.
We can do a code refactoring for show resource after this is done.
This would make the code more independent and would make the review easier

[yiguolei]

Add a system table is better solution for resources

[morningman]

LGTM

[github-actions]

PR approved by at least one committer and no changes requested.

[yiguolei]

LGTM