Uh oh!
There was an error while loading. Please reload this page.
[Feature](multimodal_embed) Support multimodal(file) embed - #62147
Conversation
hello-stephen
commented
Apr 7, 2026
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Uh oh!
There was an error while loading. Please reload this page.
linrrzqqq
commented
Apr 7, 2026
run buildall |
hello-stephen
commented
Apr 7, 2026
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
linrrzqqq
commented
Apr 7, 2026
run buildall |
hello-stephen
commented
Apr 7, 2026
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
linrrzqqq
commented
Apr 8, 2026
run buildall |
hello-stephen
commented
Apr 8, 2026
FE UT Coverage ReportIncrement line coverage |
linrrzqqq
commented
Apr 8, 2026
run beut |
linrrzqqq
commented
Apr 8, 2026
run buildall |
hello-stephen
commented
Apr 8, 2026
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
hello-stephen
commented
Apr 8, 2026
FE UT Coverage ReportIncrement line coverage |
linrrzqqq
commented
Apr 9, 2026
run beut |
linrrzqqq
commented
Apr 9, 2026
run buildall |
hello-stephen
commented
Apr 9, 2026
FE UT Coverage ReportIncrement line coverage |
hello-stephen
commented
Apr 9, 2026
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
hello-stephen
commented
Apr 9, 2026
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
PR approved by at least one committer and no changes requested. |
PR approved by anyone and no changes requested. |
linrrzqqq
commented
Apr 14, 2026
run buildall |
hello-stephen
commented
Apr 14, 2026
FE UT Coverage ReportIncrement line coverage |
zclllyybb
commented
Apr 14, 2026
/review |
There was a problem hiding this comment.
Pull request overview
This PR adds multimodal (file-based) embedding support to the EMBED AI function by accepting JSON input that can reference image/video/audio content (including S3 URIs via presigned URLs), and wires the necessary query/session metadata from FE to BE to execute these requests.
Changes:
- Extend
EMBEDto accept(resource, JSON)in FE/BE and add BE-side multimodal request building + presigned URL generation. - Track and forward only the AI resources actually used by a statement (instead of all AI resources).
- Add
file_presigned_url_ttl_secondsquery/session option and expand AI provider support (e.g., JINA) plus new/updated UTs.
Reviewed changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| regression-test/suites/ai_p0/test_ai_functions.groovy | Loosen error-message matching for AI HTTP failures in regression tests |
| gensrc/thrift/PaloInternalService.thrift | Add file_presigned_url_ttl_seconds to TQueryOptions |
| fe/fe-core/src/main/java/org/apache/doris/qe/runtime/ThriftPlansBuilder.java | Send only statement-used AI resources to BE (Nereids path) |
| fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java | Add/forward file_presigned_url_ttl_seconds session variable |
| fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java | Send only statement-used AI resources to BE (legacy/Coordinator path) |
| fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/ai/Embed.java | Add JSON signatures + register used AI resource names during analysis |
| fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/ai/AIFunction.java | Register used AI resource names for scalar AI functions |
| fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/AIAgg.java | Register used AI resource names for AI aggregates |
| fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java | Add tracking set for used AI resource names |
| fe/fe-core/src/main/java/org/apache/doris/datasource/property/constants/AIProperties.java | Add JINA as a supported provider type |
| fe/fe-core/src/main/java/org/apache/doris/catalog/Resource.java | Add helper to register used AI resource names into StatementContext |
| be/test/ai/embed_test.cpp | Add UT coverage for multimodal embed + presigned URL path |
| be/test/ai/ai_function_test.cpp | Add UT coverage for transport errors, non-200 handling, and parsing robustness |
| be/test/ai/ai_adapter_test.cpp | Add UT coverage for multimodal request building and JINA provider |
| be/src/exprs/function/ai/embed.h | Implement multimodal JSON input handling + presigned URL generation |
| be/src/exprs/function/ai/ai_similarity.h | Refactor to per-row execution + robust float parsing (fast_float) |
| be/src/exprs/function/ai/ai_functions.h | Refactor base execution flow; add HTTP status validation + embedding-request helper |
| be/src/exprs/function/ai/ai_filter.h | Refactor to per-row execution + stricter boolean parsing |
| be/src/exprs/function/ai/ai_adapter.h | Add multimodal embedding APIs and implement for multiple providers (Gemini/Qwen/Voyage/Jina) |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
| // If it's a direct http/https URL, use it as-is | ||
| if (_starts_with_ignore_case(uri, "http://") || _starts_with_ignore_case(uri, "https://")) { | ||
| media_url = uri; | ||
| return Status::OK(); | ||
| } | ||
| S3ClientConf s3_client_conf; | ||
| RETURN_IF_ERROR(init_s3_client_conf_from_json(file_input, s3_client_conf)); | ||
| auto s3_client = S3ClientFactory::instance().create(s3_client_conf); | ||
| if (s3_client == nullptr) { | ||
| return Status::InternalError("Failed to create S3 client for EMBED file input"); | ||
| } | ||
| S3URI s3_uri(uri); | ||
| RETURN_IF_ERROR(s3_uri.parse()); | ||
| std::string bucket = s3_uri.get_bucket(); | ||
| std::string key = s3_uri.get_key(); | ||
| DORIS_CHECK(!bucket.empty() && !key.empty()); | ||
| media_url = s3_client->generate_presigned_url({.bucket = bucket, .key = key}, ttl_seconds, | ||
| s3_client_conf); |
There was a problem hiding this comment.
_resolve_media_url(): any non-http(s) uri is treated as S3 and parsed with S3URI, but S3URI::parse() explicitly accepts paths without a scheme and sets bucket="". The subsequent DORIS_CHECK(!bucket.empty() && !key.empty()) can therefore abort for inputs like "path/to/file" or malformed URIs. Please require an explicit s3:// scheme (or otherwise validate the scheme) and return Status::InvalidArgument on invalid/missing bucket/key instead of DORIS_CHECK.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Found 2 blocking correctness issues.
be/src/exprs/function/ai/embed.h:_parse_file_input()treats user-provided JSON as an invariant and usesDORIS_CHECKafter parsing. BecauseEMBED(..., <json>)accepts any JSON value at the FE type level, inputs such asCAST('[]' AS JSON)orCAST('null' AS JSON)can abort the BE instead of returning a normal user-facing error.be/src/exprs/function/ai/ai_adapter.h: Gemini multimodal requests ignore the actualcontent_typefrom the JSON payload and hardcode one MIME per coarse media type. Valid JPEG/WebP/WAV/etc. inputs will therefore be sent with the wrong MIME type and can be rejected by Gemini.
Critical checkpoint conclusions:
- Goal of the task: Partially achieved. The PR adds multimodal
EMBEDplumbing, provider implementations, session-variable forwarding, and tests, but the two issues above block correctness for valid/invalid user inputs. - Small, clear, focused change: Mostly focused on multimodal embed, though it spans FE analysis/planning, thrift/query options, BE execution, adapters, and tests.
- Concurrency: No new concurrency-sensitive code path found.
- Special lifecycle / static initialization: No lifecycle or SIOF issue found.
- Configuration items:
file_presigned_url_ttl_secondsis added as a session variable and forwarded throughTQueryOptions; query-scoped propagation looks correct. - Incompatible changes / compatibility: The new thrift/query-option fields are optional, so no blocking compatibility issue was identified.
- Functionally parallel code paths: AI resource forwarding was updated in both the original planner/coordinator path and the Nereids
ThriftPlansBuilderpath. - Special conditional checks: Not sufficient in the current form. The JSON-object assumption is enforced with a crash, and Gemini MIME handling is too coarse.
- Test coverage: Happy-path coverage is decent (provider request builders, multimodal direct URL/S3 URL, regression), but there is no negative coverage for non-object JSON input or varied Gemini MIME subtypes.
- Observability: HTTP failure paths are observable, but the
DORIS_CHECKpath bypasses normal error reporting. - Transaction / persistence / data-write correctness: Not applicable for this PR.
- FE-BE variable passing: The new session variable and AI resource metadata appear to be forwarded in the relevant paths.
- Performance: No blocking performance issue found from the changed code paths.
- Other issues: None beyond the two blocking findings above.
| std::string file_json = | ||
| JsonbToJson::jsonb_to_json_string(file_column.column->get_data_at(row_num).data, | ||
| file_column.column->get_data_at(row_num).size); | ||
| file_input.Parse(file_json.c_str()); |
There was a problem hiding this comment.
_parse_file_input() is operating on user-provided JSON, but this uses DORIS_CHECK after parsing. The FE signature accepts any JSON value here, so a query like SELECT EMBED('r', CAST('[]' AS JSON)) (or CAST('null' AS JSON)) reaches this path and aborts the BE instead of returning a normal user-facing error. This should validate and return Status::InvalidArgument/Status::InternalError, not crash.
There was a problem hiding this comment.
JSON parsing requires consistency with field types in the FILE type, and should be expanded to FILE parsing.
| if (!model_name.starts_with("models/")) { | ||
| model_name = "models/" + model_name; | ||
| } | ||
| doc.AddMember("model", rapidjson::Value(model_name.c_str(), allocator), allocator); |
There was a problem hiding this comment.
This hardcodes Gemini to one MIME per coarse media type (image/png, audio/mpeg, video/mp4) and ignores the actual content_type supplied in the FILE JSON. _infer_media_type() accepts any subtype with the same prefix, so valid inputs like JPEG/WebP/WAV/etc. will be sent downstream with the wrong MIME and can be rejected or misinterpreted. Please pass the original content_type through to this request builder (or reject unsupported subtypes before signing/sending the URL).
linrrzqqq
commented
Apr 14, 2026
run cloud_p0 |
linrrzqqq
commented
Apr 14, 2026
run p0 |
hello-stephen
commented
Apr 14, 2026
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
1 similar comment
hello-stephen
commented
Apr 14, 2026
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
hello-stephen
commented
Apr 14, 2026
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
hello-stephen
commented
Apr 14, 2026
skip check_coverage |
Uh oh!
There was an error while loading. Please reload this page.
) Support image/video/audio embedding from json. Supported multimodal_embed AI provider : Gemini(image, video, audio), Qwen(image, video), Voyage(image, video), Jina(image) Supported authentication methods include IAM + EXTERNAL_ID / AK + SK ```text -- ima role, video mysql> SELECT ARRAY_SIZE( -> EMBED( -> 'qwen_mul_embed', -> CAST('{ '> "uri": "s3://selectdb-qa-test-3/lzq-multimodal-test/video/45944deac7d96c872f559f2ef94ea0a9.mp4", '> "content_type": "video/mp4", '> "provider": "AWS", '> "endpoint": "s3.us-east-1.amazonaws.com", '> "region": "us-east-1", '> "role_arn": "arn:aws:iam::447051187841:role/lzq-role-test", '> "external_id": "1001" '> }' AS JSON) -> ) -> ) AS video_embed_size; +------------------+ | video_embed_size | +------------------+ | 2560 | +------------------+ -- ima role, video mysql> SELECT ARRAY_SIZE( -> EMBED( -> 'qwen_mul_embed', -> CAST('{ '> "uri": "s3://selectdb-qa-test-3/lzq-multimodal-test/image/ai-agg.png", '> "content_type": "image/png", '> "provider": "AWS", '> "endpoint": "s3.us-east-1.amazonaws.com", '> "region": "us-east-1", '> "role_arn": "arn:aws:iam::447051187841:role/lzq-role-test", '> "external_id": "1001" '> }' AS JSON) -> ) -> ) AS img_embed_size; +----------------+ | img_embed_size | +----------------+ | 2560 | +----------------+ -- gemini multimodal embed mysql> SELECT ARRAY_SIZE( -> EMBED( -> 'gemini_mul_embed', -> CAST('{ '> "uri": "s3://selectdb-qa-test-3/lzq-multimodal-test/image/ai-agg.png", '> "content_type": "image/png", '> "provider": "AWS", '> "endpoint": "s3.us-east-1.amazonaws.com", '> "region": "us-east-1", '> "role_arn": "arn:aws:iam::447051187841:role/lzq-role-test", '> "external_id": "1001" '> }' AS JSON) -> ) -> ) AS img_embed_size; +----------------+ | img_embed_size | +----------------+ | 3072 | +----------------+ -- ak, sk Doris> SELECT ARRAY_SIZE( -> EMBED( -> 'qwen_mul_embed', -> CAST('{ '> "uri": "s3://<bucket>/<key>/test_img.png", '> "content_type": "image/png", '> "endpoint": "cos.ap-hongkong.myqcloud.com", '> "region": "ap-hongkong", '> "ak": "<ak>", '> "sk": "<sk>" '> }' AS JSON) -> ) -> ) AS img_embed_size_ak_sk; +----------------------+ | img_embed_size_ak_sk | +----------------------+ | 2560 | +----------------------+ ```
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
doc: apache/doris-website#4036
Support image/video/audio embedding from json.
Supported multimodal_embed AI provider : Gemini(image, video, audio), Qwen(image, video), Voyage(image, video), Jina(image)
Supported authentication methods include IAM + EXTERNAL_ID / AK + SK
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)