Skip to content

[test](regression) skip Arrow-incompatible complex outputs - #65182

Merged
yiguolei merged 1 commit into
apache:masterfrom
shuke987:fix-arrow-flight-skip-incompatible-cases
Jul 3, 2026
Merged

[test](regression) skip Arrow-incompatible complex outputs#65182
yiguolei merged 1 commit into
apache:masterfrom
shuke987:fix-arrow-flight-skip-incompatible-cases

Conversation

@shuke987

Copy link
Copy Markdown
Contributor

What

  • Skip map cast cases that intentionally produce null map keys when running through Arrow Flight SQL.
  • Use Arrow-specific bitmap assertions that compare bitmap_to_string(user_id) instead of raw bitmap bytes.

Why

  • Arrow Map cannot represent null keys, while the JDBC text path can render {null:...} for these cast semantics checks.
  • Raw BITMAP output is protocol-specific: JDBC renders \N, while Arrow exposes binary bytes. The stable semantic assertion is bitmap_to_string.

Testing

Not run. This PR is intentionally independent of #65084; run buildall after #65084 is merged.

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@shuke987

Copy link
Copy Markdown
ContributorAuthor

run buildall

@shuke987

Copy link
Copy Markdown
ContributorAuthor

/review

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed for PR 65182.

I did not find an evidence-backed issue that needs an inline review comment. The PR is test-only and keeps the changes focused on Arrow Flight SQL incompatibilities:

  • test_cast.groovy skips only the two map-cast checks that intentionally produce null map keys when context.useArrowFlightSql() is true.
  • test_aggregate_all_functions.groovy keeps the JDBC bitmap checks unchanged and adds Arrow-specific qt labels that compare bitmap_to_string(user_id) instead of the raw BITMAP column.
  • The added .out blocks match the projected Arrow query shape.

Critical checkpoint conclusions:

  • Goal and coverage: the patch addresses the stated Arrow Flight SQL test incompatibilities; it adds/updates regression expectations for the affected aggregate checks and skips the Arrow-incompatible map-key cases.
  • Scope and focus: the modification is small and limited to regression tests and expected output.
  • Concurrency/lifecycle/persistence/config/protocol: not applicable; no production FE/BE/cloud code or persisted format changes.
  • Parallel paths: JDBC and Arrow paths are separated by context.useArrowFlightSql(), with existing JDBC checks retained.
  • Expected output correctness: the regression framework allows skipped qt blocks by searching forward to the executed tag, so preserving the JDBC-only map_to_map2 and map_to_map3 output blocks is compatible with skipping them in Arrow mode.
  • User focus: no additional user-provided focus was supplied.

Subagent conclusions:

  • optimizer-rewrite: no candidates; final convergence returned NO_NEW_VALUABLE_FINDINGS.
  • tests-session-config: no candidates; final convergence returned NO_NEW_VALUABLE_FINDINGS.
  • No candidate became an inline comment, and no duplicate suppression was needed.

Validation:

  • Reviewed the GitHub PR file list and patch for all three changed files.
  • Refreshed the PR head before submission; head remained eea7146b5e6a6ac91b6c302ff865c1616cecbafe.
  • Checked the PR patch for added trailing whitespace and reverse-apply consistency.
  • Did not run regression tests because this checkout lacks .worktree_initialized and thirdparty/installed; validation was static.

@yiguolei
yiguolei merged commit 6dbf100 into apache:masterJul 3, 2026
32 checks passed
morningman added a commit that referenced this pull request Sep 2, 2026
…connection (#67381)
### What problem does this PR solve?
Issue Number: close#67364
Related PR: #65182, #65615
Problem Summary:
Reading anything over Arrow Flight SQL fails with `INTERNAL` /
`IllegalStateException` as soon as the query hits the FE sql cache.
The FE sql cache is keyed by `<catalog>.<db>:<user>:<sql text>`
(`NereidsSqlCacheManager.generateCacheKey`) and is shared by every
protocol, but its rows are MySQL wire protocol packets that
`StmtExecutor.sendCachedValues` replays through a `MysqlChannel`. An
Arrow Flight SQL connection has no channel, so an entry created by an
identical MySQL query makes `handleQueryStmt` take the cached-plan
branch and fail `Preconditions.checkState(connectType == MYSQL)` in
`sendFields()`. The client sees:
```
INTERNAL: get flight info statement failed, after executeQueryStatement handleQuery,
error code: ERR_UNKNOWN_ERROR, error msg: IllegalStateException, msg: null
```
The `CacheAnalyzer` branch right below it is already gated on `channel
!= null` with a `// TODO support arrow flight sql`; only this
cached-plan replay was left unguarded.
**This is not about `HLL` / `QUANTILE_STATE`.** The issue was reported
on raw aggregate-state columns, but both the FE schema helper and the BE
map `HLL` / `BITMAP` / `QUANTILE_STATE` to Arrow `binary` and have Arrow
writers for them, and they read back correctly once the query is
actually executed. They only looked special because that sql text was
the one primed through the MySQL control session; `select 1` fails
exactly the same way.
Two conditions have to line up, which is why this is not seen more
often:
1. The same sql text must have been run on a MySQL connection first (an
Arrow Flight connection never populates the cache).
2. Both sessions must agree on every session variable the cache compares
(`NereidsSqlCacheManager.usedVariablesChanged` over the
`affectQueryResult*` set). The MySQL **JDBC driver** adds
`STRICT_TRANS_TABLES` to `sql_mode` at connect time while the Arrow
Flight JDBC driver does not, so a JDBC control session masks the bug --
a `mysql` CLI session, or any client that leaves `sql_mode` alone, does
not.
### What is changed
- `ConnectProcessor.executeQuery`: look the sql cache up only for a
MySQL connection. Any other protocol re-executes the query and gets its
result from the BE. Such a connection never populates the cache either,
so this only removes a broken read path; it does not change MySQL
behaviour.
- `StmtExecutor.handleQueryStmt`: assert the channel in the cached-plan
branch, so a future regression names the protocol instead of throwing a
bare `IllegalStateException`.
- `SessionVariable`: mark `return_object_data_as_binary` as
`affectQueryResultInExecution`. It is forwarded to the BE and decides
whether the MySQL result writer serializes `HLL` / `BITMAP` /
`QUANTILE_STATE` as their raw bytes or as NULL, so it changes the cached
rows and must take part in the cache key comparison. Without it, a
session that turns it on is served the NULLs cached by a session that
had it off -- a separate, pure-MySQL-protocol correctness bug in the
same family.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@shuke987@hello-stephen@yiguolei