Skip to content

[SPARK-48792][SQL] Fix regression for INSERT with partial column list to a table with char/varchar - #47198

Closed
yaooqinn wants to merge 2 commits into
apache:masterfrom
yaooqinn:SPARK-48792
Closed

[SPARK-48792][SQL] Fix regression for INSERT with partial column list to a table with char/varchar#47198
yaooqinn wants to merge 2 commits into
apache:masterfrom
yaooqinn:SPARK-48792

Conversation

@yaooqinn

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

#41262 introduced a regression by applying literals with char/varchar type in query output for table insertions, see

https://github.com/apache/spark/pull/41262/files#diff-6e331e8f1c67b5920fb46263b6e582ec6e6a253ee45543559c9692a72a1a40ecR187-R188

This causes bugs

24/07/0316:29:01ERRORExecutor: Exceptionintask0.0instage0.0 (TID0)
org.apache.spark.SparkException: [INTERNAL_ERROR] UnsupporteddatatypeVarcharType(64). SQLSTATE: XX000atorg.apache.spark.SparkException$.internalError(SparkException.scala:92)
atorg.apache.spark.SparkException$.internalError(SparkException.scala:96)
org.apache.spark.SparkUnsupportedOperationException: VarcharType(64) isnotsupportedyet.
atorg.apache.spark.sql.errors.QueryExecutionErrors$.dataTypeUnsupportedYetError(QueryExecutionErrors.scala:993)
atorg.apache.spark.sql.execution.datasources.orc.OrcSerializer.newConverter(OrcSerializer.scala:209)
atorg.apache.spark.sql.execution.datasources.orc.OrcSerializer.$anonfun$converters$2(OrcSerializer.scala:35)
atscala.collection.immutable.List.map(List.scala:247)

Why are the changes needed?

Bugfix

Does this PR introduce any user-facing change?

no

How was this patch tested?

new tests

Was this patch authored or co-authored using generative AI tooling?

no

@yaooqinn
yaooqinn marked this pull request as draft July 3, 2024 09:32
Comment threadsql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala Outdated
@yaooqinn
yaooqinn marked this pull request as ready for review July 3, 2024 12:04
@yaooqinn

Copy link
Copy Markdown
MemberAuthor

Merged to master, thank you for the review @cloud-fan

@yaooqinn
yaooqinn deleted the SPARK-48792 branch March 27, 2026 11:44
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@yaooqinn@cloud-fan