[SPARK-58807][SQL] Preserve CHAR/VARCHAR on language surfaces under standardSemantics by srielau · Pull Request #58087 · apache/spark

srielau · 2026-08-18T18:58:12Z

What changes were proposed in this pull request?

Parent epic: SPARK-58794. Unique ticket: SPARK-58807 (CTAS/CREATE VIEW/CTE inherit CHAR/VARCHAR). Also covers SPARK-58808 (SQL FUNCTION assignment), SPARK-58809 (session variables), and SPARK-58811 (ALTER/temp/external).

#58080 (SPARK-58798 LCT) is on master. Clients (JDBC / HS2 / Connect) landed in #58132 / SPARK-58806.

Language-surface hardening under spark.sql.charVarchar.standardSemantics.enabled:

Bug fixes (format write)

ORC:CharType/VarcharType extend StringType, so write used to stamp spark.sql.catalyst.type=string and file-only inference lost the constraint. Stamp char(n) / varchar(n) via CharVarcharUtils.charVarcharTypeName. Write stays on ORC STRING plus that attribute (not native ORC CHAR/VARCHAR), so ORC maxLength does not fight Spark store assignment. Unbounded STRING, including collated STRING, still stamps string (same as Avro: collation is not a file-only round-trip in this PR).
Avro: same subclass trap on leaves, map keys, and ser/de. Stamp spark.sql.catalyst.type on STRING and spark.sql.catalyst.mapKey.type on maps. Restore only StringType subtypes; a non-string stamp is IncompatibleSchemaException. sql/core has no avro data source; ser/de is covered with DataFileWriter.

Coverage

CTAS / CREATE VIEW (CHAR and VARCHAR) / CTE inherit CHAR/VARCHAR
ALTER COLUMN equal-length CHAR/VARCHAR; V2 VARCHAR widen / CHAR to VARCHAR
Session / script variables, FETCH INTO, SQL FUNCTION params and RETURNS
EXTERNAL TABLE scan pad / overflow
CHAR/VARCHAR vs non-string compare and COALESCE
JSON / CSV with a user-specified CHAR/VARCHAR schema
ORC catalog + file-only round-trip, including cross-flag (standardSemantics on then off, first-class types off, preserveCharVarcharTypeInfo only)
Avro schema conversion, nested struct, CHAR map keys, and serializer/deserializer values

Why are the changes needed?

Without ORC/Avro catalyst-type stamping, Spark-written files cannot re-infer CHAR/VARCHAR under the flag. Language surfaces (CTAS/VIEW/CTE/vars/functions/user schemas) need explicit coverage so schema fidelity does not regress.

Does this PR introduce any user-facing change?

Yes, when the flag is on: ORC and Avro preserve CHAR/VARCHAR logical types across write/read instead of collapsing to STRING. CTAS/VIEW/CTE/vars/functions behavior matches the foundation rules and is now tested.

How was this patch tested?

BasicCharVarcharTestSuite / language surfaces (including Avro ser/de and ORC cross-flag)
Regenerated charvarchar-standard-semantics.sql goldens

srielau · 2026-08-19T19:43:33Z

Rebased onto #58080 (4e9205b, including the CHAR-pad nested-cast fix). Regenerated charvarchar-standard-semantics goldens after the LCT golden conflicts.

…esult schemas ### What changes were proposed in this pull request? Covers Connect JDBC mapping ([SPARK-58806](https://issues.apache.org/jira/browse/SPARK-58806)), plus HiveServer2 / JDBC result metadata ([SPARK-58804](https://issues.apache.org/jira/browse/SPARK-58804)) and `DatabaseMetaData.getColumns` `COLUMN_SIZE` ([SPARK-58805](https://issues.apache.org/jira/browse/SPARK-58805)). Parent: [SPARK-58794](https://issues.apache.org/jira/browse/SPARK-58794). Independent of the remaining CHAR/VARCHAR follow-ups. Rebased onto `master` after #58033; can merge without #58080 / #58087 / #58130. Unique vs master: srielau/spark@master...serge-rielau_data/SPARK-58794-clients Expose first-class `CHAR(n)` and `VARCHAR(n)` to SQL clients when `spark.sql.charVarchar.standardSemantics.enabled` is on, and let those clients decode result rows that carry the types. Metadata: - Spark Connect JDBC maps `CharType` / `VarcharType` to `java.sql.Types.CHAR` / `VARCHAR`, with Java string values and precision / display size `n`. - Connect `DatabaseMetaData.getColumns` reports `COLUMN_SIZE` as the declared character length `n`. `CHAR_OCTET_LENGTH` is the UTF-8 maximum byte capacity `4 * n` (saturating at `Int.MaxValue`); unbounded STRING stays 0. - Connect `DatabaseMetaData.getTypeInfo` lists `CHAR` and `VARCHAR` with `CREATE_PARAMS = length`. Unbounded `STRING` remains a `Types.VARCHAR` row without create params. - HiveServer2 `getColumns` reports `COLUMN_SIZE = n` for `VARCHAR(n)` as well as `CHAR(n)`, and `CHAR_OCTET_LENGTH = 4 * n` for both (null for unbounded STRING). - HiveServer2 result metadata coverage pins Hive JDBC names (`char` / `varchar`, no length) and precision `n` for CAST results. Decode (JVM Connect client): - The Connect client process has no engine `SQLConf`, so `RowEncoder.encoderFor` still follows the local `charVarcharFirstClassTypes` flag and rejects CHAR/VARCHAR when first-class types are off. - `RowEncoder.encoderForResultSchema` always accepts CHAR/VARCHAR. `SparkResult.createEncoder` uses it only for `UnboundRowEncoder` (engine-produced result schemas). Other encoder bindings keep `encoderFor`. - Arrow serializer / deserializer treat `CharEncoder` / `VarcharEncoder` like `StringEncoder`, and deserialize via `StringHelper.plainStringType` so STRING Arrow vectors up-cast to CHAR/VARCHAR. - JDBC CAST `ResultSet.next` / `collect` needs this path; metadata-only mapping is not enough. Python Connect already maps proto `char` / `var_char` in `pyspark.sql.connect.types`; a mixed classic vs Connect test covers schema and `collect()`. The existing Connect proto already carries CHAR/VARCHAR and their lengths. ### Why are the changes needed? Spark Connect JDBC rejected first-class `CharType` and `VarcharType` as unsupported because its metadata mapping only recognized the `StringType` singleton. `getColumns` hardcoded `CHAR_OCTET_LENGTH` to 0, and `getTypeInfo` listed only unbounded STRING. HiveServer2 already identified VARCHAR catalog columns but reported `COLUMN_SIZE` as unknown. Clients therefore cannot reliably describe or size CHAR/VARCHAR columns even though Catalyst retains the type and length. Separately, a server with standard semantics sends CHAR/VARCHAR in the result schema. The JVM Connect client then failed at row decode (`UNSUPPORTED_DATA_TYPE_FOR_ENCODER`) because `RowEncoder.encoderFor` read the client's default conf (`charVarcharFirstClassTypes = false`). JDBC CAST collect and Spark Connect `SparkResult` both hit that path. ### Does this PR introduce _any_ user-facing change? Yes, when first-class CHAR/VARCHAR types are present: JDBC and HiveServer2 metadata now report the corresponding JDBC type and declared character length instead of rejecting the type or reporting an unknown size. `CHAR_OCTET_LENGTH` is the UTF-8 maximum byte capacity (`4 * n`), not the character length. Connect clients can also collect CHAR/VARCHAR result columns instead of failing to decode the schema. ### How was this patch tested? - `RowEncoderSuite`: `encoderFor` still raises `UNSUPPORTED_DATA_TYPE_FOR_ENCODER` for CHAR/VARCHAR when both first-class flags are off; `encoderForResultSchema` accepts the same schema. - `ArrowEncoderSuite`: CHAR/VARCHAR Arrow round-trip (top-level, nested struct, array). - `SparkConnectJdbcDataTypeSuite`: CAST result metadata and table `getColumns` (`CHAR(4)` / `VARCHAR(6)`, `COLUMN_SIZE`, `CHAR_OCTET_LENGTH` = 16 / 24); CAST collect after decode. - `SparkConnectDatabaseMetaDataSuite`: `getTypeInfo` rows for CHAR / VARCHAR. - `SparkMetadataOperationSuite`: HiveServer2 `getColumns` / CAST result metadata (`CHAR_OCTET_LENGTH` = `4 * n`). - `pyspark.sql.tests.connect.test_connect_basic.SparkConnectBasicTests.test_char_varchar_result_schema`: classic vs Connect schema and `collect()`. Local compile of `sql-api`, `connect-client-jdbc`, and `hive-thriftserver` passed. HiveServer2 / full Connect JDBC in this environment remain blocked (`CheckReturnValue` while compiling `connect-common`; FIPS Python multiprocessing). Please treat the suites above as the merge gate. ### Was this patch authored or co-authored using generative AI tooling? Yes (Cursor). Closes#58132 from srielau/serge-rielau_data/SPARK-58794-clients. Authored-by: Serge Rielau <serge@rielau.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

…esult schemas ### What changes were proposed in this pull request? Covers Connect JDBC mapping ([SPARK-58806](https://issues.apache.org/jira/browse/SPARK-58806)), plus HiveServer2 / JDBC result metadata ([SPARK-58804](https://issues.apache.org/jira/browse/SPARK-58804)) and `DatabaseMetaData.getColumns` `COLUMN_SIZE` ([SPARK-58805](https://issues.apache.org/jira/browse/SPARK-58805)). Parent: [SPARK-58794](https://issues.apache.org/jira/browse/SPARK-58794). Independent of the remaining CHAR/VARCHAR follow-ups. Rebased onto `master` after #58033; can merge without #58080 / #58087 / #58130. Unique vs master: srielau/spark@master...serge-rielau_data/SPARK-58794-clients Expose first-class `CHAR(n)` and `VARCHAR(n)` to SQL clients when `spark.sql.charVarchar.standardSemantics.enabled` is on, and let those clients decode result rows that carry the types. Metadata: - Spark Connect JDBC maps `CharType` / `VarcharType` to `java.sql.Types.CHAR` / `VARCHAR`, with Java string values and precision / display size `n`. - Connect `DatabaseMetaData.getColumns` reports `COLUMN_SIZE` as the declared character length `n`. `CHAR_OCTET_LENGTH` is the UTF-8 maximum byte capacity `4 * n` (saturating at `Int.MaxValue`); unbounded STRING stays 0. - Connect `DatabaseMetaData.getTypeInfo` lists `CHAR` and `VARCHAR` with `CREATE_PARAMS = length`. Unbounded `STRING` remains a `Types.VARCHAR` row without create params. - HiveServer2 `getColumns` reports `COLUMN_SIZE = n` for `VARCHAR(n)` as well as `CHAR(n)`, and `CHAR_OCTET_LENGTH = 4 * n` for both (null for unbounded STRING). - HiveServer2 result metadata coverage pins Hive JDBC names (`char` / `varchar`, no length) and precision `n` for CAST results. Decode (JVM Connect client): - The Connect client process has no engine `SQLConf`, so `RowEncoder.encoderFor` still follows the local `charVarcharFirstClassTypes` flag and rejects CHAR/VARCHAR when first-class types are off. - `RowEncoder.encoderForResultSchema` always accepts CHAR/VARCHAR. `SparkResult.createEncoder` uses it only for `UnboundRowEncoder` (engine-produced result schemas). Other encoder bindings keep `encoderFor`. - Arrow serializer / deserializer treat `CharEncoder` / `VarcharEncoder` like `StringEncoder`, and deserialize via `StringHelper.plainStringType` so STRING Arrow vectors up-cast to CHAR/VARCHAR. - JDBC CAST `ResultSet.next` / `collect` needs this path; metadata-only mapping is not enough. Python Connect already maps proto `char` / `var_char` in `pyspark.sql.connect.types`; a mixed classic vs Connect test covers schema and `collect()`. The existing Connect proto already carries CHAR/VARCHAR and their lengths. ### Why are the changes needed? Spark Connect JDBC rejected first-class `CharType` and `VarcharType` as unsupported because its metadata mapping only recognized the `StringType` singleton. `getColumns` hardcoded `CHAR_OCTET_LENGTH` to 0, and `getTypeInfo` listed only unbounded STRING. HiveServer2 already identified VARCHAR catalog columns but reported `COLUMN_SIZE` as unknown. Clients therefore cannot reliably describe or size CHAR/VARCHAR columns even though Catalyst retains the type and length. Separately, a server with standard semantics sends CHAR/VARCHAR in the result schema. The JVM Connect client then failed at row decode (`UNSUPPORTED_DATA_TYPE_FOR_ENCODER`) because `RowEncoder.encoderFor` read the client's default conf (`charVarcharFirstClassTypes = false`). JDBC CAST collect and Spark Connect `SparkResult` both hit that path. ### Does this PR introduce _any_ user-facing change? Yes, when first-class CHAR/VARCHAR types are present: JDBC and HiveServer2 metadata now report the corresponding JDBC type and declared character length instead of rejecting the type or reporting an unknown size. `CHAR_OCTET_LENGTH` is the UTF-8 maximum byte capacity (`4 * n`), not the character length. Connect clients can also collect CHAR/VARCHAR result columns instead of failing to decode the schema. ### How was this patch tested? - `RowEncoderSuite`: `encoderFor` still raises `UNSUPPORTED_DATA_TYPE_FOR_ENCODER` for CHAR/VARCHAR when both first-class flags are off; `encoderForResultSchema` accepts the same schema. - `ArrowEncoderSuite`: CHAR/VARCHAR Arrow round-trip (top-level, nested struct, array). - `SparkConnectJdbcDataTypeSuite`: CAST result metadata and table `getColumns` (`CHAR(4)` / `VARCHAR(6)`, `COLUMN_SIZE`, `CHAR_OCTET_LENGTH` = 16 / 24); CAST collect after decode. - `SparkConnectDatabaseMetaDataSuite`: `getTypeInfo` rows for CHAR / VARCHAR. - `SparkMetadataOperationSuite`: HiveServer2 `getColumns` / CAST result metadata (`CHAR_OCTET_LENGTH` = `4 * n`). - `pyspark.sql.tests.connect.test_connect_basic.SparkConnectBasicTests.test_char_varchar_result_schema`: classic vs Connect schema and `collect()`. Local compile of `sql-api`, `connect-client-jdbc`, and `hive-thriftserver` passed. HiveServer2 / full Connect JDBC in this environment remain blocked (`CheckReturnValue` while compiling `connect-common`; FIPS Python multiprocessing). Please treat the suites above as the merge gate. ### Was this patch authored or co-authored using generative AI tooling? Yes (Cursor). Closes#58132 from srielau/serge-rielau_data/SPARK-58794-clients. Authored-by: Serge Rielau <serge@rielau.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 6af294e) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

…tandardSemantics Stamp catalyst type attributes for ORC/Avro write so constrained types round-trip, and cover CTAS/VIEW/ALTER/variables/SQL FUNCTION/JSON/CSV/ORC under the flag.

Extend language-surface tests for VARCHAR session vars, compound local vars, cursor FETCH INTO, and SQL FUNCTION parameters under standardSemantics.

Cover trailing-blank trim on SET/local SET/FETCH, STRING-into-CHAR FETCH, CHAR SQL FUNCTION params, and RETURNS VARCHAR under standardSemantics.

…on V2 Document that V1 file-source ALTER rejects length changes; assert V2 CheckAnalysis allows VARCHAR widen and CHAR->VARCHAR with first-class types.

…mantics Pad on scan, write assignment, and oversize scan error for CREATE EXTERNAL TABLE over file data. Hive TRANSFORM is out of scope for this epic.

Assert CHAR and VARCHAR follow STRING for comparisons and COALESCE against numeric, datetime, and boolean types, including padded CHAR vs the same bytes.

srielau

SQL Language review (`b971a2a`)

This is the right split for SPARK-58807 / 58808 / 58809 / 58811: keep CHAR/VARCHAR first-class on language surfaces, and fix the StringType subclass trap on ORC/Avro write so file inference can restore char(n) / varchar(n).

Language-surface tests are stronger than the PR description suggests. CTAS, VIEW, variables, compound SET, FETCH, SQL FUNCTION params/RETURNS, EXTERNAL TABLE, and V2 ALTER all assert values (pad / blank-trim / overflow), not only typeof. Avro case (_: StringType, STRING) is representation-safe: CHAR/VARCHAR/collated STRING are still UTF8. Parquet is not missing; it already stores the Spark schema. Flag-off writes should not stamp CHAR because replace-to-STRING happens before the format writer.

Findings

Medium -- Avro still matches map keys against the StringType singleton. Leaf CHAR/VARCHAR are handled, but MapType(StringType, ...) and kt == StringType reject CHAR/VARCHAR keys. toAvroType throws IncompatibleSchemaException; this is the same trap this PR is fixing, and toAvroTypeWithDefaults is the RocksDB state-store path.
Medium -- catalyst-type stamping is copied in three places (toAvroType, toAvroTypeWithDefaults, OrcUtils.orcTypeDescription), each of which must stay before generic StringType. A shared CharVarcharUtils verb for the logical type string would make the next subclass harder to miss.
Medium -- Avro ser/de is untested end-to-end. The suite only round-trips SchemaConverters. AvroUtils.prepareWrite uses toAvroType, and this PR also widens serializer/deserializer. A file write/read (or at least a nested field + map-key case) would pin the changed paths.
Medium -- no cross-flag file contract. Please write with standardSemantics=true and read with it off, and write with first-class types off and confirm CHAR is not stamped. Distinguish standardSemantics from preserveCharVarcharTypeInfo.
Low -- ORC writes STRING + Spark attribute, not native ORC CHAR/VARCHAR. That matches TimeType/interval stamping and avoids ORC-layer maxLength fighting Spark store assignment. Please document that choice next to the match; toCatalystSchema already claims native char/varchar can round-trip on read.
Low -- PR description is behind the commits. Compare/COALESCE vs non-string, EXTERNAL TABLE scan overflow, FETCH/scripting, and Avro ser/de widening are in the patch but not in the summary. CTE inheritance is claimed; the goldens only cover CTAS/VIEW (CHAR-only view).

Not blocking on (5)/(6). (1) is the same class of bug the ORC/Avro leaf matches exist to fix.

srielau

SQL Language re-review (`340a730`)

Previous mediums are addressed:

Avro map keys:MapType(kt: StringType, ...) plus spark.sql.catalyst.mapKey.type on the map schema is the right shape (Avro keys have no independent schema). Serializer/deserializer match the same StringType subclass. toAvroTypeWithDefaults uses avroMapSchema, so the RocksDB path is not left behind.
Shared stamp verb:SparkCharVarcharUtils.charVarcharTypeName plus avroStringSchema / avroMapSchema collapses the three copy-paste CHAR/VARCHAR arms.
Avro ser/de: DataFileWriter round-trip in sql/core is a fair substitute for the connector data source.
Cross-flag ORC: write-on/read-off, write-off (no CHAR stamp), and preserveCharVarcharTypeInfo are covered.
ORC STRING vs native CHAR: the comment next to the match is the documentation we asked for.
CTE / VARCHAR view: goldens and the Scala suite both pin them.

parseStampedStringType is the right defensive check for physical STRING.

Remaining (non-blocking)

Low -- Avro DataFile round-trip only asserts field 0. Nested VARCHAR and the CHAR map key are converted in the schema assert, but deserialize is only checked for c. Please assert the struct field and map entry as well so the widened ser/de arms cannot regress silently.
Low -- PR description is still the pre-follow-up text. It still says "Avro schema conversion unit round-trip" and does not mention map-key stamping, CTE/VARCHAR view goldens, or the ORC STRING+attr choice.
Low -- ORC unbounded STRING now stamps s.typeName. The old case _: StringType used StringType.typeName (string). Collated unbounded STRING can now round-trip on file-only ORC reads; Avro still does not stamp collation on unbounded STRING. If that ORC change is intentional, a one-line test would pin it; if not, use charVarcharTypeName(s).getOrElse(StringType.typeName) for the unbounded arm.

No new architectural issues. Not blocking on the lows.

…G uncollated

cloud-fan

0 blocking, 0 non-blocking, 3 nits.
The implementation and coverage look sound; only three test-comment wording fixes remain.

Nits: 3 minor items (see inline comments).

Verification

Static verification: traced OrcUtils.orcTypeDescription from its StringType match through setAttribute and the StringType.typeName fallback, then reviewed the Avro/ORC cross-flag, nested/map-key, CTE/view, and golden coverage. I did not run tests.

…tandardSemantics ### What changes were proposed in this pull request? Parent epic: [SPARK-58794](https://issues.apache.org/jira/browse/SPARK-58794). Unique ticket: [SPARK-58807](https://issues.apache.org/jira/browse/SPARK-58807) (CTAS/CREATE VIEW/CTE inherit CHAR/VARCHAR). Also covers [SPARK-58808](https://issues.apache.org/jira/browse/SPARK-58808) (SQL FUNCTION assignment), [SPARK-58809](https://issues.apache.org/jira/browse/SPARK-58809) (session variables), and [SPARK-58811](https://issues.apache.org/jira/browse/SPARK-58811) (ALTER/temp/external). #58080 (SPARK-58798 LCT) is on master. Clients (JDBC / HS2 / Connect) landed in #58132 / SPARK-58806. Language-surface hardening under `spark.sql.charVarchar.standardSemantics.enabled`: **Bug fixes (format write)** - **ORC:** `CharType`/`VarcharType` extend `StringType`, so write used to stamp `spark.sql.catalyst.type=string` and file-only inference lost the constraint. Stamp `char(n)` / `varchar(n)` via `CharVarcharUtils.charVarcharTypeName`. Write stays on ORC `STRING` plus that attribute (not native ORC CHAR/VARCHAR), so ORC `maxLength` does not fight Spark store assignment. Unbounded STRING, including collated STRING, still stamps `string` (same as Avro: collation is not a file-only round-trip in this PR). - **Avro:** same subclass trap on leaves, map keys, and ser/de. Stamp `spark.sql.catalyst.type` on STRING and `spark.sql.catalyst.mapKey.type` on maps. Restore only `StringType` subtypes; a non-string stamp is `IncompatibleSchemaException`. sql/core has no avro data source; ser/de is covered with `DataFileWriter`. **Coverage** - CTAS / CREATE VIEW (CHAR and VARCHAR) / CTE inherit CHAR/VARCHAR - ALTER COLUMN equal-length CHAR/VARCHAR; V2 VARCHAR widen / CHAR to VARCHAR - Session / script variables, FETCH INTO, SQL FUNCTION params and RETURNS - EXTERNAL TABLE scan pad / overflow - CHAR/VARCHAR vs non-string compare and COALESCE - JSON / CSV with a user-specified CHAR/VARCHAR schema - ORC catalog + file-only round-trip, including cross-flag (`standardSemantics` on then off, first-class types off, `preserveCharVarcharTypeInfo` only) - Avro schema conversion, nested struct, CHAR map keys, and serializer/deserializer values ### Why are the changes needed? Without ORC/Avro catalyst-type stamping, Spark-written files cannot re-infer CHAR/VARCHAR under the flag. Language surfaces (CTAS/VIEW/CTE/vars/functions/user schemas) need explicit coverage so schema fidelity does not regress. ### Does this PR introduce _any_ user-facing change? Yes, when the flag is on: ORC and Avro preserve CHAR/VARCHAR logical types across write/read instead of collapsing to STRING. CTAS/VIEW/CTE/vars/functions behavior matches the foundation rules and is now tested. ### How was this patch tested? - `BasicCharVarcharTestSuite` / language surfaces (including Avro ser/de and ORC cross-flag) - Regenerated `charvarchar-standard-semantics.sql` goldens Closes#58087 from srielau/SPARK-58794-surfaces. Authored-by: Serge Rielau <serge@rielau.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit db6fc3a) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

cloud-fan · 2026-08-22T14:28:29Z

Merge Summary:

merged into master db6fc3a
merged into branch-4.x 85bc102

Posted by merge_spark_pr.py

This was referenced Aug 19, 2026
[SPARK-58794][SQL] Promote CHAR/VARCHAR to STRING via ImplicitTypeCasts #58130
Closed
[SPARK-58806][SQL][CONNECT] Expose CHAR/VARCHAR metadata and decode result schemas #58132
Closed

srielau force-pushed the SPARK-58794-surfaces branch 2 times, most recently from 5e45d90 to 6dc3909Compare August 19, 2026 19:43

srielau force-pushed the SPARK-58794-surfaces branch 4 times, most recently from 27a9fd9 to 4935142Compare August 19, 2026 23:13

srielau added 7 commits August 20, 2026 17:01

[SPARK-58794][SQL] Preserve CHAR/VARCHAR on language surfaces under s…
b2ff5e2
…tandardSemantics Stamp catalyst type attributes for ORC/Avro write so constrained types round-trip, and cover CTAS/VIEW/ALTER/variables/SQL FUNCTION/JSON/CSV/ORC under the flag.

[SPARK-58794][SQL] Cover session/script/cursor CHAR/VARCHAR surfaces
e61e1eb
Extend language-surface tests for VARCHAR session vars, compound local vars, cursor FETCH INTO, and SQL FUNCTION parameters under standardSemantics.

[SPARK-58794][SQL] Fill CHAR/VARCHAR assignment-cast coverage gaps
992d73f
Cover trailing-blank trim on SET/local SET/FETCH, STRING-into-CHAR FETCH, CHAR SQL FUNCTION params, and RETURNS VARCHAR under standardSemantics.

[SPARK-58794][SQL] Cover ALTER VARCHAR widen under standardSemantics …
85b8899
…on V2 Document that V1 file-source ALTER rejects length changes; assert V2 CheckAnalysis allows VARCHAR widen and CHAR->VARCHAR with first-class types.

[SPARK-58794][SQL] Cover EXTERNAL TABLE CHAR/VARCHAR under standardSe…
4eb97bc
…mantics Pad on scan, write assignment, and oversize scan error for CREATE EXTERNAL TABLE over file data. Hive TRANSFORM is out of scope for this epic.

[SPARK-58794][SQL] Cover CHAR/VARCHAR vs non-string compare and COALESCE
226e91e
Assert CHAR and VARCHAR follow STRING for comparisons and COALESCE against numeric, datetime, and boolean types, including padded CHAR vs the same bytes.

[SPARK-58794][SQL] Regenerate CHAR/VARCHAR goldens after LCT rebase
b971a2a

srielau force-pushed the SPARK-58794-surfaces branch from 4935142 to b971a2aCompare August 20, 2026 17:04

srielau changed the title ~~[SPARK-58794][SQL] Preserve CHAR/VARCHAR on language surfaces under standardSemantics~~[SPARK-58807][SQL] Preserve CHAR/VARCHAR on language surfaces under standardSemanticsAug 20, 2026

srielau commented Aug 20, 2026
View reviewed changes

[SPARK-58807][SQL] Stamp Avro CHAR map keys and cover format round-trips
340a730

srielau commented Aug 20, 2026
View reviewed changes

Comment threadsql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
Comment threadsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala Outdated

srielau added 2 commits August 20, 2026 17:56

[SPARK-58807][SQL] Assert nested Avro CHAR/VARCHAR and keep ORC STRIN…
011ade6
…G uncollated

[SPARK-58807][SQL] Retrigger CI
d192eaa

cloud-fan approved these changes Aug 21, 2026
View reviewed changes

Comment threadsql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala Outdated
Comment threadsql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala Outdated
Comment threadsql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala Outdated

srielau added 2 commits August 21, 2026 20:45

[SPARK-58807][SQL] Fix CHAR/VARCHAR test comment wording
1bc95d0

[SPARK-58807][SQL] Scalafmt SparkCharVarcharUtils docstring
2e0dfa5

cloud-fan closed this in db6fc3aAug 22, 2026

Navigation Menu

Uh oh!

[SPARK-58807][SQL] Preserve CHAR/VARCHAR on language surfaces under standardSemantics - #58087

[SPARK-58807][SQL] Preserve CHAR/VARCHAR on language surfaces under standardSemantics#58087
srielau wants to merge 12 commits into
apache:masterfrom
srielau:SPARK-58794-surfaces

srielau commented Aug 18, 2026•
edited
Loading

Uh oh!

srielau commented Aug 19, 2026

Uh oh!

srielau left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

srielau left a comment

Uh oh!

Uh oh!

Uh oh!

cloud-fan left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cloud-fan commented Aug 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

srielau commented Aug 18, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

srielau commented Aug 19, 2026

Uh oh!

srielau left a comment

Choose a reason for hiding this comment

SQL Language review (b971a2a)

Findings

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

srielau left a comment

Choose a reason for hiding this comment

SQL Language re-review (340a730)

Remaining (non-blocking)

Uh oh!

Uh oh!

Uh oh!

cloud-fan left a comment

Choose a reason for hiding this comment

Verification

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cloud-fan commented Aug 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

srielau commented Aug 18, 2026•
edited
Loading

SQL Language review (`b971a2a`)

SQL Language re-review (`340a730`)