[Feature] Support Doris Catalog by Chen768959 · Pull Request #55274 · apache/doris

Chen768959 · 2025-08-25T16:58:37Z

The new Doris Catalog replaces the previous method of accessing external tables in remote Doris clusters via JDBC Catalog.

related: #56011

It retrieves metadata of Doris external tables through HTTP APIs.
The metadata information for Doris external tables is more accurate, fully capturing various metadata from remote cluster tables, such as primary keys, bucketing keys, and native Doris data types.
The underlying implementation uses Arrow Flight SQL to communicate with the remote Doris cluster, achieving approximately 4 times higher transmission performance compared to the single-node JDBC Catalog.
It supports concurrent retrieval of Arrow response results from the remote Doris cluster, with transmission performance scaling linearly as the cluster size increases.

hello-stephen · 2025-08-25T16:58:42Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Copilot

Pull Request Overview

This PR introduces support for Doris Catalog, a new feature that enables efficient access to external tables in remote Doris clusters using Arrow Flight SQL protocol. The implementation replaces the previous JDBC-based approach with an HTTP API-based metadata retrieval system combined with Arrow Flight SQL for data transmission, providing approximately 4x performance improvement.

Key changes:

Implements new Doris Catalog with HTTP-based metadata retrieval and Arrow Flight SQL data access
Adds comprehensive test suite for all data types, statistics, refresh operations, and predicate pushdown
Introduces configuration support for remote Doris cluster connections

Reviewed Changes

Copilot reviewed 47 out of 47 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
regression-test/suites/external_table_p0/remote_doris/*.groovy	Comprehensive test suite for remote Doris functionality
fe/fe-core/src/main/java/org/apache/doris/datasource/doris/*	Core implementation of remote Doris catalog, database, table, and REST client
be/src/vec/exec/format/table/remote_doris_reader.*	Backend Arrow Flight reader implementation
gensrc/thrift/*.thrift	Thrift definitions for remote Doris table types
fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/*	REST API endpoints for schema information

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

morningman · 2025-10-24T03:45:31Z

run buildall

doris-robot · 2025-10-24T04:11:00Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	80.64% (1649/2045)
Line Coverage	67.01% (29109/43437)
Region Coverage	67.37% (14430/21420)
Branch Coverage	57.73% (7675/13294)

doris-robot · 2025-10-24T07:00:56Z

BE UT Coverage Report

Increment line coverage 0.00% (0/75) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.61% (17940/34099)
Line Coverage	37.85% (162779/430030)
Region Coverage	32.23% (123978/384671)
Branch Coverage	33.67% (54370/161478)

morningman · 2025-11-01T15:55:21Z

run buildall

github-actions · 2025-11-01T15:56:56Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-11-01T15:56:58Z

PR approved by anyone and no changes requested.

doris-robot · 2025-11-01T16:26:28Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	80.63% (1673/2075)
Line Coverage	67.00% (29517/44052)
Region Coverage	67.50% (14665/21725)
Branch Coverage	57.77% (7797/13496)

doris-robot · 2025-11-01T19:14:42Z

BE UT Coverage Report

Increment line coverage 0.00% (0/75) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.68% (18050/34263)
Line Coverage	37.98% (164024/431914)
Region Coverage	32.30% (125010/387086)
Branch Coverage	33.72% (54699/162224)

hello-stephen · 2025-11-10T08:45:16Z

BE Regression && UT Coverage Report

Increment line coverage 10.13% (8/79) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.47% (24268/33957)
Line Coverage	57.99% (252583/435530)
Region Coverage	53.47% (211125/394826)
Branch Coverage	54.66% (89884/164449)

hello-stephen · 2025-11-10T09:02:35Z

FE Regression Coverage Report

Increment line coverage 64.10% (300/468) 🎉
Increment coverage report
Complete coverage report

The new Doris Catalog replaces the previous method of accessing external tables in remote Doris clusters via JDBC Catalog. related: apache#56011 1. It retrieves metadata of Doris external tables through HTTP APIs. 2. The metadata information for Doris external tables is more accurate, fully capturing various metadata from remote cluster tables, such as primary keys, bucketing keys, and native Doris data types. 3. The underlying implementation uses Arrow Flight SQL to communicate with the remote Doris cluster, achieving approximately 4 times higher transmission performance compared to the single-node JDBC Catalog. 4. It supports concurrent retrieval of Arrow response results from the remote Doris cluster, with transmission performance scaling linearly as the cluster size increases.

…58002) Fix memory alignment issues in read_column_from_arrow by using memcpy This PR addresses potential memory alignment problems when reading data from Arrow arrays in the `read_column_from_arrow` method. Instead of directly casting potentially unaligned pointers, we now use `memcpy` to safely copy data into properly aligned memory, preventing misaligned address runtime errors. Followup #55274

…pache#58002) Fix memory alignment issues in read_column_from_arrow by using memcpy This PR addresses potential memory alignment problems when reading data from Arrow arrays in the `read_column_from_arrow` method. Instead of directly casting potentially unaligned pointers, we now use `memcpy` to safely copy data into properly aligned memory, preventing misaligned address runtime errors. Followup apache#55274

The new Doris Catalog replaces the previous method of accessing external tables in remote Doris clusters via JDBC Catalog. related: apache#56011 1. It retrieves metadata of Doris external tables through HTTP APIs. 2. The metadata information for Doris external tables is more accurate, fully capturing various metadata from remote cluster tables, such as primary keys, bucketing keys, and native Doris data types. 3. The underlying implementation uses Arrow Flight SQL to communicate with the remote Doris cluster, achieving approximately 4 times higher transmission performance compared to the single-node JDBC Catalog. 4. It supports concurrent retrieval of Arrow response results from the remote Doris cluster, with transmission performance scaling linearly as the cluster size increases.

…pache#58002) Fix memory alignment issues in read_column_from_arrow by using memcpy This PR addresses potential memory alignment problems when reading data from Arrow arrays in the `read_column_from_arrow` method. Instead of directly casting potentially unaligned pointers, we now use `memcpy` to safely copy data into properly aligned memory, preventing misaligned address runtime errors. Followup apache#55274

…58248) 1. Fix for unaligned memory in arrow MapArray parsing. 2. arrow Map unaligned memory ut. 3. arrow Struct unaligned memory ut. Followup #55274

The new Doris Catalog replaces the previous method of accessing external tables in remote Doris clusters via JDBC Catalog. related: apache#56011 1. It retrieves metadata of Doris external tables through HTTP APIs. 2. The metadata information for Doris external tables is more accurate, fully capturing various metadata from remote cluster tables, such as primary keys, bucketing keys, and native Doris data types. 3. The underlying implementation uses Arrow Flight SQL to communicate with the remote Doris cluster, achieving approximately 4 times higher transmission performance compared to the single-node JDBC Catalog. 4. It supports concurrent retrieval of Arrow response results from the remote Doris cluster, with transmission performance scaling linearly as the cluster size increases.

…pache#58248) 1. Fix for unaligned memory in arrow MapArray parsing. 2. arrow Map unaligned memory ut. 3. arrow Struct unaligned memory ut. Followup apache#55274

…58248) 1. Fix for unaligned memory in arrow MapArray parsing. 2. arrow Map unaligned memory ut. 3. arrow Struct unaligned memory ut. Followup #55274

The new Doris Catalog replaces the previous method of accessing external tables in remote Doris clusters via JDBC Catalog. related: apache#56011 1. It retrieves metadata of Doris external tables through HTTP APIs. 2. The metadata information for Doris external tables is more accurate, fully capturing various metadata from remote cluster tables, such as primary keys, bucketing keys, and native Doris data types. 3. The underlying implementation uses Arrow Flight SQL to communicate with the remote Doris cluster, achieving approximately 4 times higher transmission performance compared to the single-node JDBC Catalog. 4. It supports concurrent retrieval of Arrow response results from the remote Doris cluster, with transmission performance scaling linearly as the cluster size increases.

bp #55274 #58002 #58248 --------- Co-authored-by: Chen768959 <67011523+Chen768959@users.noreply.github.com>

…pache#58002) Fix memory alignment issues in read_column_from_arrow by using memcpy This PR addresses potential memory alignment problems when reading data from Arrow arrays in the `read_column_from_arrow` method. Instead of directly casting potentially unaligned pointers, we now use `memcpy` to safely copy data into properly aligned memory, preventing misaligned address runtime errors. Followup apache#55274

…pache#58248) 1. Fix for unaligned memory in arrow MapArray parsing. 2. arrow Map unaligned memory ut. 3. arrow Struct unaligned memory ut. Followup apache#55274

…pache#58002) Fix memory alignment issues in read_column_from_arrow by using memcpy This PR addresses potential memory alignment problems when reading data from Arrow arrays in the `read_column_from_arrow` method. Instead of directly casting potentially unaligned pointers, we now use `memcpy` to safely copy data into properly aligned memory, preventing misaligned address runtime errors. Followup apache#55274

…pache#58248) 1. Fix for unaligned memory in arrow MapArray parsing. 2. arrow Map unaligned memory ut. 3. arrow Struct unaligned memory ut. Followup apache#55274

Chen768959 requested review from CalvinKirs and morningman as code owners August 25, 2025 16:58

Chen768959 mentioned this pull request Aug 25, 2025
[future](doris catalog) Support the new Doris catalog #50611
Closed

Chen768959 force-pushed the dev_doris_catalog_chen_v2 branch from c2f6c22 to 530a97dCompare August 26, 2025 16:13

zy-kkk self-assigned this Aug 28, 2025

Chen768959 force-pushed the dev_doris_catalog_chen_v2 branch 3 times, most recently from 32fd79c to f1ee51bCompare September 18, 2025 13:26

zy-kkk requested a review from Copilot September 19, 2025 02:23

CopilotAI reviewed Sep 19, 2025
View reviewed changes

Chen768959 force-pushed the dev_doris_catalog_chen_v2 branch 5 times, most recently from 6d04d1d to 9bfbf71Compare September 28, 2025 07:57

morningman force-pushed the dev_doris_catalog_chen_v2 branch from 9bfbf71 to 190b529Compare October 23, 2025 09:22

morningman force-pushed the dev_doris_catalog_chen_v2 branch from 190b529 to 6311b1dCompare November 1, 2025 15:55

morningman added the dev/4.0.x label Nov 1, 2025

morningman previously approved these changes Nov 1, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Nov 1, 2025

github-actionsBot added the reviewed label Nov 1, 2025

morningman merged commit 53ebb5c into apache:masterNov 10, 2025
27 of 28 checks passed

github-actionsBot added the dev/4.0.x-conflict label Nov 10, 2025

morningman mentioned this pull request Nov 19, 2025
[fix](datatype)Fix unaligned memory access in read_column_from_arrow #58002
Merged

morningman mentioned this pull request Nov 20, 2025
branch-4.0: [Feature] Support Doris Catalog (#55274) #58162
Merged

morningman mentioned this pull request Nov 22, 2025
[fix](datatype)Fix for unaligned memory in arrow MapArray parsing. #58248
Merged

yiguolei pushed a commit that referenced this pull request Nov 24, 2025
branch-4.0: [Feature] Support Doris Catalog (#55274) (#58162)
7989173
bp #55274 #58002 #58248 --------- Co-authored-by: Chen768959 <67011523+Chen768959@users.noreply.github.com>

yiguolei added dev/4.0.2-merged and removed dev/4.0.x dev/4.0.x-conflict labels Nov 24, 2025

morningman mentioned this pull request Jul 15, 2026
[Tracking] Arrow Flight SQL and ADBC Catalog: issues and improvements #65615
Open
63 tasks

Uh oh!

Conversation

Chen768959 commented Aug 25, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hello-stephen commented Aug 25, 2025

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

morningman commented Oct 24, 2025

Uh oh!

doris-robot commented Oct 24, 2025

Cloud UT Coverage Report

Uh oh!

doris-robot commented Oct 24, 2025

BE UT Coverage Report

Uh oh!

morningman commented Nov 1, 2025

Uh oh!

github-actionsBot commented Nov 1, 2025

Uh oh!

github-actionsBot commented Nov 1, 2025

Uh oh!

doris-robot commented Nov 1, 2025

Cloud UT Coverage Report

Uh oh!

doris-robot commented Nov 1, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Nov 10, 2025

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Nov 10, 2025

FE Regression Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Chen768959 commented Aug 25, 2025•
edited
Loading