Skip to content

branch-4.1: [chore](external) Upgrade Hive shade and align BE exec version#66907

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yiguolei merged 1 commit into from
Aug 19, 2026
Merged

branch-4.1: [chore](external) Upgrade Hive shade and align BE exec version #66907

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions be/src/agent/be_exec_version_manager.cpp
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -126,11 +126,13 @@ void BeExecVersionManager::check_function_compatibility(int current_be_exec_vers
// a. use new fixed object serialization way.
// 11: start from master
// a. enforce Iceberg SQL MERGE cardinality only when every executing BE supports it.
// 12: start from Doris 4.1
// 12: start from master
// a. support Variant columns and delete-only writer omission in Iceberg SQL MERGE.
// 13: start from Doris 4.1
// a. support strict ownership hash routing for external table sink writers.
// b. support Paimon default fixed-bucket routing in the external sink exchange.

const int BeExecVersionManager::max_be_exec_version = 12;
const int BeExecVersionManager::max_be_exec_version = 13;
const int BeExecVersionManager::min_be_exec_version = 0;
std::map<std::string, std::set<int>> BeExecVersionManager::_function_change_map {};
std::set<std::string> BeExecVersionManager::_function_restrict_map;
Expand Down
2 changes: 1 addition & 1 deletion be/src/agent/be_exec_version_manager.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -34,7 +34,7 @@ constexpr inline int USE_CONST_SERDE =
8; // support const column in serialize/deserialize function: PR #41175
constexpr inline int USE_NEW_FIXED_OBJECT_SERIALIZATION_VERSION = 10;
constexpr inline int SUPPORT_ICEBERG_MERGE_CARDINALITY_VERSION = 11;
constexpr inline int SUPPORT_EXTERNAL_TABLE_SINK_HASH_VERSION = 12;
constexpr inline int SUPPORT_EXTERNAL_TABLE_SINK_HASH_VERSION = 13;

class BeExecVersionManager {
public:
Expand Down
2 changes: 1 addition & 1 deletion fe/fe-common/src/main/java/org/apache/doris/common/Config.java
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -2260,7 +2260,7 @@ public class Config extends ConfigBase {
* Max data version of backends serialize block.
*/
@ConfField(mutable = false)
public static int max_be_exec_version = 12;
public static int max_be_exec_version = 13;

/**
* Min data version of backends serialize block.
Expand Down
83 changes: 24 additions & 59 deletions fe/fe-core/pom.xml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -468,32 +468,6 @@ under the License.
<groupId>com.google.flatbuffers</groupId>
<artifactId>flatbuffers-java</artifactId>
</dependency>
<!-- Must precede hive-catalog-shade: that published shade artifact embeds
Paimon 1.3 classes, while Doris compiles against paimon.version. -->
<dependency>
<groupId>org.apache.paimon</groupId>
<artifactId>paimon-core</artifactId>
</dependency>
<dependency>
<groupId>org.apache.paimon</groupId>
<artifactId>paimon-common</artifactId>
</dependency>
<dependency>
<groupId>org.apache.paimon</groupId>
<artifactId>paimon-hive-connector-3.1</artifactId>
</dependency>
<dependency>
<groupId>org.apache.paimon</groupId>
<artifactId>paimon-format</artifactId>
</dependency>
<dependency>
<groupId>org.apache.paimon</groupId>
<artifactId>paimon-s3</artifactId>
</dependency>
<dependency>
<groupId>org.apache.paimon</groupId>
<artifactId>paimon-jindo</artifactId>
</dependency>
<dependency>
<groupId>org.apache.doris</groupId>
<artifactId>hive-catalog-shade</artifactId>
Expand DownExpand Up@@ -604,6 +578,30 @@ under the License.
<artifactId>iceberg-aws</artifactId>
<version>${iceberg.version}</version>
</dependency>
<dependency>
<groupId>org.apache.paimon</groupId>
<artifactId>paimon-core</artifactId>
</dependency>

<dependency>
<groupId>org.apache.paimon</groupId>
<artifactId>paimon-common</artifactId>
</dependency>

<dependency>
<groupId>org.apache.paimon</groupId>
<artifactId>paimon-format</artifactId>
</dependency>

<dependency>
<groupId>org.apache.paimon</groupId>
<artifactId>paimon-s3</artifactId>
</dependency>
<dependency>
<groupId>org.apache.paimon</groupId>
<artifactId>paimon-jindo</artifactId>
</dependency>

<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>glue</artifactId>
Expand DownExpand Up@@ -1117,44 +1115,11 @@ under the License.
<overWriteSnapshots>false</overWriteSnapshots>
<overWriteIfNewer>true</overWriteIfNewer>
<includeScope>runtime</includeScope>
<!-- hive-catalog-shade 3.1.1 embeds Paimon 1.3. The FE supplies
Paimon from the managed paimon.version instead. Repackage this
dependency below without its embedded Paimon runtime. -->
<excludeArtifactIds>hive-catalog-shade</excludeArtifactIds>
<skip>${skip.plugin}</skip>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<version>3.1.0</version>
<executions>
<execution>
<id>package-hive-catalog-shade-without-paimon</id>
<phase>package</phase>
<configuration>
<target>
<delete dir="${project.build.directory}/hive-catalog-shade-without-paimon"/>
<mkdir dir="${project.build.directory}/hive-catalog-shade-without-paimon"/>
<unzip src="${settings.localRepository}/org/apache/doris/hive-catalog-shade/${doris.hive.catalog.shade.version}/hive-catalog-shade-${doris.hive.catalog.shade.version}.jar"
dest="${project.build.directory}/hive-catalog-shade-without-paimon">
<patternset>
<exclude name="org/apache/paimon/**"/>
<exclude name="META-INF/services/org.apache.paimon*"/>
</patternset>
</unzip>
<jar destfile="${project.build.directory}/lib/hive-catalog-shade-${doris.hive.catalog.shade.version}.jar"
basedir="${project.build.directory}/hive-catalog-shade-without-paimon"/>
</target>
</configuration>
<goals>
<goal>run</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>cobertura-maven-plugin</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion fe/pom.xml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -226,7 +226,7 @@ under the License.
<module>fe-authentication</module>
</modules>
<properties>
<doris.hive.catalog.shade.version>3.1.1</doris.hive.catalog.shade.version>
<doris.hive.catalog.shade.version>3.1.2</doris.hive.catalog.shade.version>
<!-- iceberg 1.9.1 depends avro on 1.12 -->
<avro.version>1.12.1</avro.version>
<parquet.version>1.17.0</parquet.version>
Expand Down
Loading