Release Note 1.2.2

[morningman]

Release Note 1.2.1

New Features

• Multi-Catalog
  • Support automatic synchronization of Hive metastore
    • https://doris.apache.org/docs/dev/lakehouse/multi-catalog/#metadata-update
  • Support reading the Iceberg Snapshot, and viewing the Snapshot history
    • https://doris.apache.org/docs/dev/lakehouse/multi-catalog/iceberg
  • JDBC Catalog supports PostgreSQL, Clickhouse, Oracle, SQLServer
    • https://doris.apache.org/docs/dev/lakehouse/multi-catalog/jdbc
  • JDBC Catalog supports Insert operation
    • https://doris.apache.org/docs/dev/lakehouse/multi-catalog/jdbc/
• New Functions
  • width_bucket：https://doris.apache.org/docs/dev/sql-manual/sql-functions/width-bucket

Behavior Changes

• Disable BE's page cache by default: disable_storage_page_cache=true

  Turn off this configuration to optimize memory usage and reduce the risk of memory OOM.
  But it will reduce the query latency of some small queries.
  If you are sensitive to query latency, or have high concurrency and small query scenarios, you can configure disable_storage_page_cache=false to enable page cache again.

• Add new session variable group_by_and_having_use_alias_first, used to control whether the group and having clauses use alias

  • https://doris.apache.org/docs/dev/advanced/variables

Improvement

• Compaction
  • Support Vertical Compaction. To optimize the compaction overhead and efficiency of wide tables.
    • https://doris.apache.org/docs/dev/advanced/best-practice/compaction
  • Support Segment Compaction. Fix -238 and -235 issues with high frequency imports.
    • https://doris.apache.org/docs/dev/advanced/best-practice/compaction
• Multi-Catalog
  • Hive Catalog can be compatible with Hive version 1/2/3
    • https://doris.apache.org/docs/dev/lakehouse/multi-catalog/hive#hive-versions
  • Hive Catalog can access JuiceFS based HDFS with Broker.
    • https://doris.apache.org/docs/dev/lakehouse/multi-catalog/hive
  • Iceberg Catalog Support Hive Metastore and Rest Catalog type.
    • https://doris.apache.org/docs/dev/lakehouse/multi-catalog/iceberg
  • ES Catalog support _id column mapping。
    • https://doris.apache.org/docs/dev/lakehouse/multi-catalog/es
• Support auto bucket when creating table
  • TODO: Doc
• Optimize Iceberg V2 read performance with large number of delete rows.
• Support for reading Iceberg tables after Schema Evolution
• Parquet Reader handles column name case correctly.
• Support for accessing Hadoop KMS-encrypted HDFS.
  • https://doris.apache.org/docs/dev/lakehouse/multi-catalog/hive
• Support to cancel the Export export task in progress.
  • https://doris.apache.org/docs/dev/sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/CANCEL-EXPORT
• Optimize the performance of explode_split with 1x.
• Optimize the read performance of nullable columns with 3x.
• Optimize some problems of Memtracker, improve memory management accuracy, and optimize memory application.

Bug Fixes

• Fixed memory leak when loading data with Doris Flink Connector [Fix](load) Fix memory leak for stream load 2pc #16430
• Fixed the possible thread scheduling problem of BE and reduce the Fragment sent timeout error caused by BE thread exhaustion [fix](vresultsink) BufferControlBlock may block all fragment handle threads  #16231
• Fixed various correctness and precision issues of column type datetimev2/decimalv3.
• Fixed the problem data correctness issue with Unique Key Merge-on-Read table.
• Fixed various known issues with the Light Schema Change feature.
• Fixed various data correctness issues of bitmap type Runtime Filter.
• Fixed the problem of poor reading performance of csv reader introduced in version 1.2.1. [fix](csv-reader) fix new csv reader's performance issue #15581
• Fixed the problem of BE OOM caused by Spark Load data download phase. [Enhancement](SparkLoad): avoid BE OOM in push task #15620
• Fixed possible metadata compatibility issues when upgrading from version 1.1 to version 1.2. [fix](meta) fix priv table load bug when upgrading to 1.2.x #15706
• Fixed the metadata problem when creating JDBC Catalog with Resource. [fix](multi-catalog) fix image loading failture when create catalog with resource #15692
• Fixed the problem of high CPU usage caused by load operation [enhancement](load) avoid schema copy to reduce cpu usage #16034
• Fixed the problem of FE OOM caused by a large number of failed Broker Load jobs. [enhancement](memory) reduce memory usage for failed broker loads #15895
• Fixed the problem of precision loss when loading floating-point types [Enhance](fix)use std::from_chars to do str cast to float/double to avoid lose precision #16190
• Fixed the problem of memory leak when useing 2PC stream load [Fix](load) Fix memory leak for stream load 2pc #16430

Other

• Add metrics to view the total rowset and segment numbers on BE
  • doris_be_all_rowset_nums and doris_be_all_segment_nums
  • https://doris.apache.org/zh-CN/docs/dev/admin-manual/maint-monitor/monitor-metrics/metrics

Big Thanks

Thanks all who contribute to this release:

@adonis0147
@AshinGau
@BePPPower
@BiteTheDDDDt
@ByteYue
@caiconghui
@cambyzju
@chenlinzhong
@DarvenDuan
@dataroaring
@Doris-Extras
@dutyu
@englefly
@freemandealer
@Gabriel39
@HappenLee
@Henry2SS
@htyoung
@isHuangXin
@JackDrogon
@jacktengg
@Jibing-Li
@kaka11chen
@Kikyou1997
@Lchangliang
@LemonLiTree
@liaoxin01
@liqing-coder
@luozenglin
@morningman
@morrySnow
@mrhhsg
@nextdreamblue
@qidaye
@qzsee
@spaces-X
@stalary
@starocean999
@weizuo93
@wsjz
@xiaokang
@xinyiZzz
@xy720
@yangzhg
@yiguolei
@yixiutt
@Yukang-Lian
@Yulei-Yang
@zclllyybb
@zddr
@zhangstar333
@zhannngchen
@zy-kkk

Full commits can be seen here

[dutyu]

The doc's url of Support to cancel the Export export task in progress. may be not right (failed to open in my pc), it should be
https://doris.apache.org/docs/dev/sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/CANCEL-EXPORT ?

[morningman]

https://doris.apache.org/docs/dev/sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/CANCEL-EXPORT

It works in my env, have you tried to clean the cookie?

[stalary]

https://doris.apache.org/en/docs/dev/lakehouse/multi-catalog/jdbc is not found.

[dutyu]

https://doris.apache.org/docs/dev/sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/CANCEL-EXPORT

It works in my env, have you tried to clean the cookie?

Yeah, this url works, but that one not works, it has an extra "/en/" part ("https://doris.apache.org/en/").

[Accelarator]

c61be0c 这个不发吗

[morningman]

c61be0c 这个不发吗

#16074 fixed this issue, and pr with dev/1.2.2-merged is included in this release

[morningman]

https://doris.apache.org/en/docs/dev/lakehouse/multi-catalog/jdbc is not found.

fixed