Doris Roadmap 2022

[morningman]

[morningman]

The following is the Roadmap for the Doris community in 2022.
The plan includes all aspects of code features, documentation, community building, etc. that are to be developed, have already been developed, and have been completed but require ongoing optimization.

The plan is currently under discussion, so if you have comments or suggestions on any aspect of the plan or beyond, please feel free to leave a comment or send an email to dev@doris.apache.org.

We will gradually create issues or jira for each direction of the plan to describe and track the progress in detail. Developers who wish to contribute are also welcome to create issues directly and associate with them (just leave a comment)

The directions marked (Good First Issue) in the plan are more independent modules, which are more suitable for newbie tasks or developers who are new to Doris. If you are interested in the relevant direction, please contact us at dev@doris.apache.org or under this issue, and we will provide detailed guidance, help and discussion.

The directions marked with (Q1) are the current work to be completed in the first quarter of 2022. We will update the schedule and progress of other directions gradually.

The marked (Done & Optimizing) directions are the directions that are currently completed but need continuous optimization. Such as ease of use, feature additions, and documentation additions.

We encourage developers to discuss anything in the dev mailing list, to subscribe to the mailing list please refer to How to subscribe.

Features

• [Roadmap] New query optimizer #7571

  • Extensible new query optimizer framework
  • [Proposal] Collect richer statistical information #6370
  • Standard test set support and performance enhancements
    + TPC-DS feature pass rate 100%
    + TPC-H performance enhancements

• [Roadmap] Push-based Query Execution Engine #7572

  • Pipeline execution engine
  • Algorithm Concurrency Control and Resource Control

• [Roadmap] Support complex type #7573

  • [Roadmap] Support Array Type（Q1） #7570
  • Map
  • Struct

• [Roadmap] Support semi-structured data #7574

  Provides Schemaless semantics for fast analysis of semi-structured data.

  • Json

• [Roadmap] Support hot and cold data separation #7575 (Q1)

  Supports cold data storage to object storage at partition granularity with remote access capabilities and local Cache acceleration.

• [Roadmap] Support multi table materialized view #7503

  Doris' current "materialized view" is more of a "materialized index" concept. Doris will later implement a true Materialized View to support full and incremental construction of single and multi-table views.

• [Roadmap] Support new data model for update #7576

  Provide Kudu-like data update support.

• [Roadmap] Support for more advanced analysis functions #7577

  • WindowFunnel Window funnel #8485

• [Roadmap] Support new UDF framework #7578

  Support for the new UDF framework has solved the problems of high writing difficulty, poor isolation, and poor compatibility with existing C++ frameworks.

  • UDF [feature] Support udf through RPC #7519
  • UDAF [Feature] Support UDAF in RPC framework #8312
  • UDTF
  • Java UDF [Feature] Support Java UDF #8389

• [Roadmap] Support more import data formats #7579 (Good First Issue)

  • [Proposal] Support common csv #7552
  • [Proposal] Support avro format load #7650

• Add more resource limits

  • [Feature] Limit cluster resource usage in user granularity #7129

• More builtin function support

  • [Feature] max_by, min_by aggregate function support #7678

Performance Optimization

• [Roadmap] Support vectorized query engine #7580 (Q1)

  • Query layer vectorization
  • Storage level vectorization
  • Vectorization function supplementation
  • Query layer storage layer arithmetic unification
  • Import Vectorization

• Json Parsing Optimization (Good First Issue)

  • [Proposal] Replace rapidjson with simdjson #7551

• [Roadmap] Compaction optimization #7743

  Optimize the performance of compaction task. And try to refactor the compaction logic. For example, only one replica do the compaction and sync to other replicas.

Stability and Observability

• [Proposal] Memory tracker optimization #7553 (Q1)

  Solve the problems of inaccurate memory prediction and OOM, and improve memory observability by global + thread + task level memory management.

• [Roadmap] Implement IO manager #7581

  Provides fine-grained IO speed limit, priority scheduling, etc. through global IO management.

• [Roadmap] Enhanced system observability #7582

  Introduces OpenTelemetry to enhance system internal state observability and unify monitoring data format.

Testing

• [Roadmap] Implementing a better unit testing framework #7583

  • FE

    Refine the FE single test framework to support multi-node simulation testing of features.

  • BE

    Provide testing framework to simplify the difficulty of writing complex unit tests (e.g. data builds) for BE.

• [Roadmap] Implementing a scalable regression testing framework #7584

  Provide Case collection or submission framework for refining and accumulating regression test sets.

• [Roadmap] Implement benchmark framework #7585

  Provide a Benchmark testing framework to ensure that adding new code does not impact performance.

• [Roadmap] Chaos test #7586

  Implement ChaosMesh chaos testing to improve the correctness and stability of the system in case of anomalies.

Functional Optimization

• [Feature] Z-Order Implement #7149

  • [Feature] Z-Order Implement #7149
  • Support vectorization engine Z-Order
  • Agg/Uniq Key model support for Z-Order
  • Schema Change

• Lateral View

  • Support bitmap, string, json_array expansion (Done & Optimizing)
  • Array type expansion support
  • Table Function

• Other features

  • [Feature] Data export function improvements #7680
  • CreateTableAsStmt support decimal

Deployment and Maintenance

• [Roadmap] Implement Doris manager #7591 (Q1)

  Provides a visual interface for Doris deployment, monitoring, and operations maintenance. Simplifies Doris deployment, scaling, upgrades, task management, status checking, and other operations.

• [Roadmap] Doris on K8S #7587 (Good First Issue)

  Provides a Helm Charts-based K8S deployment solution.

Peripheral Ecology

• [Roadmap] Multi Catalog and query federation #7588

  • [Feature] Support query hive table  #6568 (Done & Optimizing)

  • [Feature] Support create Iceberg external table #7389 (Q1)

  • Hudi

  • Parquet File Format Support

    Generate and read Parquet files directly to facilitate Doris data to be read directly by external systems.

• Data Integration

  • Routine Load support for Canal format parsing (Q1)
  • Flink Connector (Done & Optimizing)
  • Spark Connector (Done & Optimizing)
  • SeaTunnel Sink (Done & Optimizing)
  • SeaTunnel Source
  • DataX (Addax) (Done & Optimizing)
  • [Roadmap] Support automatic table structure transformation and pseudo data generation #7781

• Compilation Tools

  • [Feafure] support build thirdparty and BE using clang #7590 (Q1)

Community

• [Roadmap] Doris official website #7589 (Q1)

  Refactored Doris official website to provide best practices, community progress, blog posts, FAQ, and more.

• Doris Documentation (Good First Issue)

  Non-code contributions are as important as code contributions, and the community is very open to developers improving and proofreading the project documentation.

  • [Roadmap] Refactor all grammar help documents #6336
  • Reorganization of the official Doris documentation to improve readability, operability, and guidance.
  • Translation and proofreading of the English documentation.

• Github Action (Good First Issue)

  Introduced more Github Actions to help improve the management of the code base. This includes but is not limited to PR autoresponders, tagging, etc. If you have a good Action to recommend, please leave a comment.

[yiguolei]

For regression test and performance test, we could follow clickhouse's test method. If it is allowed, I could do this.

[yiguolei]

Clang compile is already on process, see #7451

[EmmyMiao87]

Could you please open an email to discuss Roadmap 2022 of Doris ?

[yangzhg]

支持parquet 文件存储格式也应该加进去吧

[wangshuo128]

希望考虑跨版本升级功能。

[Henry2SS]

What about supporting AVRO format in LOAD function?

[zbtzbtzbt]

Looking forward to push based pipeline engine @morningman-cmy @yiguolei

[hf200012]

Doris Manager：
1.Follow-up Doris Manager upgrade
2.User UI interaction improvement
3.Doris Manager supports Doris automated upgrade

[924060929]

我们公司已经有一个回归测试框架。大体是用groovy的dsl去完成测试sql、stream load、安装tpch等功能，大概使用方式如下图。
后续可以提给社区

[jackwener]

既然后续有这么多内容，关于社区部分建一个 RFC 目录挺有必要的，大型的 PR 的 design doc 放进去，一方面是为了社区新人的快速融入，另外也减小PR review的压力