[Proposal] Support of LIST partition

[qidaye]

List partition

Doris currently only supports Range partitioning, where data is usually partitioned by time columns.

However, in some scenarios, users want to partition by some enumerated values of columns, such as by city, etc.

Design

To add support for List partitioning, the following functional points need to be considered.

1. Support for List partition syntax in creating table statements.
2. Support for adding and deleting List partition syntax.
3. Support for List partitioning in various load operations.
4. Support for List partition pruning during query.

List partitioned tables do not need to consider dynamic partitioning.

Detailed design

Syntax

The main changes involved here include.

1. Implementation of the subclass ListPartitionDesc of the parsing class PartitionDesc
2. Implementation of metadata class PartitionInfo subclass ListPartitionInfo
3. Support for parsing and checking ListPartitionDesc in CreateTableStmt
4. Support for the creation of List Partition tables in Catalog class.
5. Metadata persistence-related changes.

The syntax is referenced from MySQL and Oracle

Single partition column

CREATE TABLE tb1 (
 k1 int, k2 varchar(128), k3 int, v1 int, v2 int
)
PARTITION BY LIST(k1)
(
 PARTITION p1 VALUES IN ("1", "3", "5"),
 PARTITION p2 VALUES IN ("2", "4", "6"),
 ...
)
...
;

Multi-partition columns

CREATE TABLE tb2 (
 k1 int, k2 varchar(128), k3 int, v1 int, v2 int
)
PARTITION BY LIST(k1, k2)
(
 PARTITION p1 VALUES IN (("1", "beijing"), ("1", "shanghai")),
 PARTITION p2 VALUES IN (("2", "beijing"), ("2", "shanghai"), ("2", "tianjin")),
 PARTITION p3 VALUES IN (("3", "beijing")),
 ...
)
...
;

NOTE: Each partition needs to ensure that the partition values are unique.

Add partition

ALTER TABLE tb1 ADD PARTITION p4 VALUES IN ("7", "8", "9");

ALTER TABLE tb2 ADD PARTITION p4 VALUES IN (("4", "tianjin"));

Load

The current load methods of Doris include Stream Load, INSERT, Routine Load, Broker Load, Hadoop Load, Spark Load.

Among them, Stream Load, INSERT, Routine Load, and Broker Load all use TabletSink class for data distribution. Our first phase supports List partition support for these load operations.

The main changes involved include:

1. Changes related to the Descriptors.TOlapTablePartitionParam structure in the Thrift structure TOlapTableSink
2. Changes related to the OlapTablePartition object in the OlapTableSink class on the BE side.

Query

The query mainly needs to implement the List Partition pruning function.

The main changes involved include:

1. Implementing the subclass ListPartitionPruner of PartitionPruner

Partition related

Support operations related to partitioned tables, such as recover, truncate, temporary partition, restore, replace, etc.

[HappenLee]

Nice Job~~

[moonming]

Please send the proposal to the mailing list too

[qidaye]

Please send the proposal to the mailing list too

Already sent to dev@doris.apache.org