[SPARK-58794][SQL] Add standardSemantics foundation for CHAR/VARCHAR by srielau · Pull Request #58033 · apache/spark

srielau · 2026-08-16T18:26:40Z

What changes were proposed in this pull request?

This is the foundation for SQL standard-aligned CHAR(n) / VARCHAR(n) support. It adds a single gate, spark.sql.charVarchar.standardSemantics.enabled (default false), under which CHAR/VARCHAR behave as first-class types with proper length semantics instead of being silently annotated STRING.

The change is best read as a path a CHAR/VARCHAR value travels through the engine, hardened one stage at a time:

Typing - CHAR/VARCHAR may now appear as first-class types in schemas and plans, rather than being erased to STRING during analysis.
Introduction (CAST) - CAST / try_cast to CHAR(n) / VARCHAR(n) keep the target type and enforce its length: CHAR is padded, VARCHAR is trimmed, and an oversized value fails with EXCEED_LIMIT_LENGTH (after trailing blanks are trimmed).
Combination (least common type) - COALESCE, CASE, UNION, IN, and friends widen along CHAR -> VARCHAR -> STRING, taking max(n, m) for the length, so combining constrained values yields the tightest type that fits all of them.
Transformation - functions that rewrite string content (upper, substr, concat / ||, regexp_*, split, mask, reverse, hex, array_join, str_to_map, json_tuple, ...) return plain STRING, because their result length is not the input's length. This is implemented as a promotion in ImplicitTypeCasts (CHAR/VARCHAR -> STRING where a plain string is expected, analogous to SHORT -> INT); the few expressions that bypass implicit casting derive the same result type directly. A registry-wide inventory test fails if any function outside an explicit pass-through allowlist returns a constrained type under the flag, so future expressions cannot silently leak a length.
Persistence (scan / write) - write-side checks preserve the declared type, and scans pad CHAR back to its length and reject oversized values, so what is read back matches what the schema promises.
Delivery (Thrift/JDBC) - RowSetUtils no longer wraps CHAR/VARCHAR (or collated STRING) results in quotes. Its fast path matched only the default-collation STRING singleton, so these types fell through to a rendering path that quoted them ("ab" instead of ab). This fix is intentionally not gated on the flag, since collated strings were mis-rendered regardless.

spark.sql.preserveCharVarcharTypeInfo remains the pre-existing experimental path; only standardSemantics applies the transformation-returns-STRING rule and the hardening above.

Out of scope / follow-up

This PR is the engine foundation only. The feature is not complete without the work below; each item is intentionally deferred so this change stays reviewable.

Clients

HiveServer2 / JDBC ResultSetMetaData: report CHAR/VARCHAR (with precision n) instead of generic STRING / VARCHAR with unbounded precision.
DatabaseMetaData.getColumns: fix COLUMN_SIZE = 0 for VARCHAR.
Spark Connect JDBC type mapping for CHAR/VARCHAR + precision.

Language surfaces

Schema fidelity for CTAS / CREATE VIEW / CTEs, ALTER, temp tables, and external file tables.
SQL FUNCTION parameters / RETURNS, and session variables (DECLARE / SET).
Format round-trips (Parquet / ORC / Avro / CSV) that preserve the logical type end to end.

Parameterized lengths (CHAR(:n) / VARCHAR(:n)) are covered in #58080 (parser already binds markers in length position; that PR pins the behavior under the flag).

Known engine gap (addressed in follow-up)

Combining two collated CHARs of different lengths but the same collation used to resolve to an indeterminate collation. Fixed and covered, along with compare / IN / set-op LCT behavior, in [SPARK-58798][SQL] Fix collated CHAR LCT and cover compare/IN/set-ops #58080.

Explicitly not in this effort

Deprecating preserveCharVarcharTypeInfo.
Iceberg CHAR/VARCHAR metadata.
Full ISO operator-result typing for every string operator.

Why are the changes needed?

Without this gate, CHAR/VARCHAR are effectively aliases for STRING: CAST stringifies the target, combinations always widen to STRING, and length is neither enforced on write nor restored on read. Standard-compliant CHAR/VARCHAR need a consistent, opt-in path through the whole engine.

Does this PR introduce any user-facing change?

Yes, when spark.sql.charVarchar.standardSemantics.enabled is true (default remains false):

CAST AS CHAR/VARCHAR returns a typed result and enforces length.
COALESCE / CASE / UNION may return CHAR/VARCHAR.
Content-transforming string functions return STRING.
Scans pad CHAR and reject oversized values.

Independently of the flag, Thrift/JDBC clients no longer see spurious quotes around CHAR/VARCHAR and collated STRING column values.

How was this patch tested?

New cases in CharVarcharTestSuite / BasicCharVarcharTestSuite / FileSourceCharVarcharTestSuite covering CAST, least-common-type, transformation, createDataFrame, and scan pad/oversize.
Golden file charvarchar-standard-semantics.sql (plus analyzer results).
A parity matrix asserting the fixed-point and single-pass analyzers agree under the flag.
A registry-wide inventory test that fails if any function outside the pass-through allowlist returns CHAR/VARCHAR.
RowSetUtilsSuite for unquoted rendering of CHAR, VARCHAR, and collated STRING; verified it fails when the RowSetUtils fix is reverted.

Introduce spark.sql.charVarchar.standardSemantics.enabled and wire first-class types, real CAST, LCT, R1 STRING results, write-side typing, and read-side pad/oversize checks (SPARK-58796 through SPARK-58801).

…dence - Preserve collationId when tightestCommonString returns CHAR/VARCHAR. - Apply R1 (transforming string functions return plain STRING) to regexp_replace/regexp_extract/regexp_extract_all/split and mask. - Make charVarchar.standardSemantics take precedence over the legacy charVarcharAsString flag in cast replacement and read-side padding. - addPaddingForScan falls back to attr.dataType when the raw-type metadata is absent, so first-class CHAR/VARCHAR attributes still get enforced. - Add tests for LCT collation, R1 on regexp/mask/split, and the preserveCharVarcharTypeInfo vs standardSemantics flag matrix.

…low-ups - Add charvarchar-standard-semantics.sql covering CAST/LCT/R1/UNION/IN/scan, including try_cast, regexp/mask/split, and nested types. - Preserve declared collation Option in tightestCommonString so default LCT still renders as char(n)/varchar(n) (not collate UTF8_BINARY). - Revert attr.dataType fallback in addPaddingForScan: metadata is the not-yet-padded marker required for ApplyCharTypePadding idempotence. - Warn once when readSideCharPadding=false under standardSemantics; document that read-side checks are identical to write-side by design. - Pin dual-run Analyzer++ parity coverage for CAST/LCT/R1/UNION.

…, binding policy Three CI failures from the standardSemantics foundation: - Guard the multi-byte VARCHAR test's CJK literals with scalastyle:off/on nonascii, matching the convention used by the variant and collation suites. - Stop the Thrift server from wrapping CHAR/VARCHAR values in quotes. The column-oriented fast path in RowSetUtils matches only the default-collation StringType singleton, so first-class CHAR/VARCHAR (and collated strings) fall through to the generic branch that calls toHiveString with nested = true. Treat string types like the geospatial types already handled there. - Give spark.sql.charVarchar.standardSemantics.enabled a binding policy. It is PERSISTED: the flag decides which types a view body resolves to, so a view created under standard semantics keeps computing CHAR/VARCHAR no matter who calls it, the same reasoning that applies to ANSI mode.

cloud-fan

1 blocking, 1 non-blocking, 2 nits.
The semantic foundation is coherent, but the newly activated RowEncoder path must preserve declared collations before merge; the remaining items are local cleanup and documentation fixes.

Correctness (1)

Blocking: sql/api/src/main/scala/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala:94: Preserve the full constrained string type in RowEncoder so collated CHAR/VARCHAR schemas retain their declared collation. -- see inline

Suggestions (1)

Non-blocking: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala:366: Short-circuit the disabled leaf validation before traversing and materializing each leaf output. -- see inline

Nits: 2 minor items (see inline comments).

Verification

I traced the opt-in predicate from SQLConf through schema admission, casts/LCT, expression result typing, dual analyzers, and data-source scan padding. The scan helpers converge on the existing write-side checks, while raw-type metadata prevents repeated padding. I also traced RowEncoder schema construction and confirmed that its leaf encoder data types reconstruct StructFields, exposing the collation loss described below.

… RowEncoder Pass the full CharType/VarcharType into CharEncoder/VarcharEncoder (like GeographyEncoder) so createDataFrame retains a declared collation instead of rebuilding a default-collation type from length alone. Also short-circuit the CheckAnalysis leaf guard, fix the VARCHAR read-side javadoc link, and rephrase a scan-padding test comment.

srielau · 2026-08-17T21:49:59Z

/trigger all

Covers SPARK-58802: prove the fixed-point and single-pass analyzers agree under spark.sql.charVarchar.standardSemantics.enabled (D19). The single-pass resolver has no Char/Varchar-specific logic -- it inherits Expression.dataType and shared TypeCoercion -- so parity is verification of the foundation code rather than a separate change, and the matrix belongs with the code it proves. Adds dual-run assertions for CAST/try_cast, LCT (COALESCE/CASE/NULL/IN/ UNION/INTERSECT), R1 transforming expressions including regexp/mask/split, nested CHAR types, collated CAST, and bare column references from tables.

…ardSemantics Allowing CharType/VarcharType in createDataFrame, Encoders.CHAR/VARCHAR, UDF return types, Dataset.to, and DataFrameReader/DataStreamReader schemas is unlocked by this PR's change to failIfHasCharVarchar, which now consults charVarcharFirstClassTypes instead of rejecting these types outright. Write-side pad/trim/EXCEED_LIMIT_LENGTH then follows from CharEncoder and CatalystTypeConverters. These tests cover those API surfaces end-to-end under spark.sql.charVarchar.standardSemantics.enabled, and extend RowEncoderSuite / UDFSuite coverage beyond the preserve-only path. They belong with the gate change rather than in a separate PR.

CharType and VarcharType extend StringType, so ImplicitTypeCasts left their length constraints in place at string-typed call sites. That broke same-type unification (overlay, string_agg, listagg) and Right's RuntimeReplaceable literals, and let reverse/hex/array_join declare CHAR(n) for longer results. Promote constrained string types to plain STRING where a string is expected (ANSI and non-ANSI), fix those three dataType overrides for R1, and type Right's literal branches after Substring's R1 result. Pass-through / LCT sites that expect AnyDataType still keep CHAR/VARCHAR.

srielau

SQL Language review

Verdict: request changes (0 Critical / 3 High / 3 Medium / 2 Low)

Solid, carefully gated foundation: charVarcharFirstClassTypes vs standardSemantics vs legacy charVarcharAsString precedence is mostly consistent; CAST/LCT/padding/encoder/dual-run coverage is strong; default-off killswitch looks intact for the Spark 4.0 path.

Please address the High items before merge — especially not shipping a golden that encodes broken collated LCT typeof, and closing the clear str_to_map R1 leak (which also illustrates that the scattered R1 approach will keep missing ExpectsInputTypes producers). The Thrift quoting change is likely correct but needs explicit acknowledgment and tests because it is not flag-gated.

Findings not attachable to the diff

High — StringToMap R1 leak (complexTypeCreator.scala):
StringToMap still does MapType(first.dataType, first.dataType) and uses ExpectsInputTypes, so R1 never applies. Under standardSemantics, str_to_map on CHAR/VARCHAR should yield map<string,string> (keys/values are parsed pieces, not the input constraint). Please strip with StringHelper.transformingStringResultType (and the ArrayBasedMapBuilder types) and add a golden/typeof + dual-run test.

Medium — test coverage gaps: Foundation tests are strong for CAST/LCT/upper/concat/dual-run. Please also cover str_to_map, Hive scan padding under the flag, and the Thrift CHAR/collated quoting path.

Low — duplicated flag precedence:charVarcharAsString && !standardSemantics appears in both CAST rewriting and ApplyCharTypePadding. A tiny conf helper would keep them in lockstep.

Recommended actions

Quarantine or fix the collated CHAR coalesce golden (string collate null).
Fix StringToMap R1 + test; sketch a longer-term central R1 mechanism / inventory test.
Document + test the ungated Thrift collated-string quoting change (or split it out).
Override Empty2Null.dataType so R1 is not inherited from String2StringExpression.
Expand Hive/Thrift/str_to_map coverage; fix the Reverse comment (Low).

Prior cloud-fan items (encoder collation, CheckAnalysis order, codegen javadoc, test wording) look addressed.

…ventory test Addresses review feedback: - str_to_map and json_tuple take their input through ExpectsInputTypes, so no implicit cast strips the length: both returned entries typed CHAR(n)/VARCHAR(n) even though the values are pulled out of the input rather than being it. - Empty2Null shared String2StringExpression's R1 result type, but empty-to-null is pass-through for every non-empty value, so it now keeps the child type. - Add an inventory test that sweeps the function registry under standardSemantics and fails when any function outside an explicit pass-through allowlist returns a constrained type. - Add RowSetUtilsSuite covering the unquoted rendering of CHAR, VARCHAR and collated string results. - Drop the goldened "string collate null" for a mixed-length collated CHAR coalesce rather than normalize it; CollationTypeCoercion reads differing lengths as a collation mismatch, which predates this change and is tracked separately.

cloud-fan

4 addressed, 0 remaining, 1 new to this AI review. (1 newly introduced, 0 late catches, 0 previously raised.)
0 blocking, 0 non-blocking, 1 nit.
The earlier review issues are addressed; one minor test-comment accuracy issue remains.

Nits: 1 minor item (see inline comments).

Verification

I traced the enabled and legacy gates through common-string typing, implicit coercion, encoder construction, scan padding/length enforcement, and Thrift rendering. I also reconciled all prior feedback against the current tree and reviewed the SQL, dual-analyzer, encoder, scan, UDF, and delivery coverage.

…shapes

### What changes were proposed in this pull request? This is the foundation for SQL standard-aligned `CHAR(n)` / `VARCHAR(n)` support. It adds a single gate, `spark.sql.charVarchar.standardSemantics.enabled` (default `false`), under which `CHAR`/`VARCHAR` behave as first-class types with proper length semantics instead of being silently annotated `STRING`. The change is best read as a path a `CHAR`/`VARCHAR` value travels through the engine, hardened one stage at a time: 1. **Typing** - `CHAR`/`VARCHAR` may now appear as first-class types in schemas and plans, rather than being erased to `STRING` during analysis. 2. **Introduction (`CAST`)** - `CAST` / `try_cast` to `CHAR(n)` / `VARCHAR(n)` keep the target type and enforce its length: `CHAR` is padded, `VARCHAR` is trimmed, and an oversized value fails with `EXCEED_LIMIT_LENGTH` (after trailing blanks are trimmed). 3. **Combination (least common type)** - `COALESCE`, `CASE`, `UNION`, `IN`, and friends widen along `CHAR -> VARCHAR -> STRING`, taking `max(n, m)` for the length, so combining constrained values yields the tightest type that fits all of them. 4. **Transformation** - functions that rewrite string content (`upper`, `substr`, `concat` / `||`, `regexp_*`, `split`, `mask`, `reverse`, `hex`, `array_join`, `str_to_map`, `json_tuple`, ...) return plain `STRING`, because their result length is not the input's length. This is implemented as a promotion in `ImplicitTypeCasts` (`CHAR`/`VARCHAR` -> `STRING` where a plain string is expected, analogous to `SHORT` -> `INT`); the few expressions that bypass implicit casting derive the same result type directly. A registry-wide inventory test fails if any function outside an explicit pass-through allowlist returns a constrained type under the flag, so future expressions cannot silently leak a length. 5. **Persistence (scan / write)** - write-side checks preserve the declared type, and scans pad `CHAR` back to its length and reject oversized values, so what is read back matches what the schema promises. 6. **Delivery (Thrift/JDBC)** - `RowSetUtils` no longer wraps `CHAR`/`VARCHAR` (or collated `STRING`) results in quotes. Its fast path matched only the default-collation `STRING` singleton, so these types fell through to a rendering path that quoted them (`"ab"` instead of `ab`). This fix is intentionally not gated on the flag, since collated strings were mis-rendered regardless. `spark.sql.preserveCharVarcharTypeInfo` remains the pre-existing experimental path; only `standardSemantics` applies the transformation-returns-`STRING` rule and the hardening above. ### Out of scope / follow-up This PR is the **engine foundation** only. The feature is not complete without the work below; each item is intentionally deferred so this change stays reviewable. **Clients** - HiveServer2 / JDBC `ResultSetMetaData`: report `CHAR`/`VARCHAR` (with precision `n`) instead of generic `STRING` / `VARCHAR` with unbounded precision. - `DatabaseMetaData.getColumns`: fix `COLUMN_SIZE = 0` for `VARCHAR`. - Spark Connect JDBC type mapping for `CHAR`/`VARCHAR` + precision. **Language surfaces** - Schema fidelity for `CTAS` / `CREATE VIEW` / CTEs, `ALTER`, temp tables, and external file tables. - SQL `FUNCTION` parameters / `RETURNS`, and session variables (`DECLARE` / `SET`). - Format round-trips (Parquet / ORC / Avro / CSV) that preserve the logical type end to end. Parameterized lengths (`CHAR(:n)` / `VARCHAR(:n)`) are covered in #58080 (parser already binds markers in length position; that PR pins the behavior under the flag). **Known engine gap (addressed in follow-up)** - Combining two collated `CHAR`s of different lengths but the same collation used to resolve to an indeterminate collation. Fixed and covered, along with compare / `IN` / set-op LCT behavior, in #58080. **Explicitly not in this effort** - Deprecating `preserveCharVarcharTypeInfo`. - Iceberg `CHAR`/`VARCHAR` metadata. - Full ISO operator-result typing for every string operator. ### Why are the changes needed? Without this gate, `CHAR`/`VARCHAR` are effectively aliases for `STRING`: `CAST` stringifies the target, combinations always widen to `STRING`, and length is neither enforced on write nor restored on read. Standard-compliant `CHAR`/`VARCHAR` need a consistent, opt-in path through the whole engine. ### Does this PR introduce _any_ user-facing change? Yes, when `spark.sql.charVarchar.standardSemantics.enabled` is `true` (default remains `false`): - `CAST AS CHAR/VARCHAR` returns a typed result and enforces length. - `COALESCE` / `CASE` / `UNION` may return `CHAR`/`VARCHAR`. - Content-transforming string functions return `STRING`. - Scans pad `CHAR` and reject oversized values. Independently of the flag, Thrift/JDBC clients no longer see spurious quotes around `CHAR`/`VARCHAR` and collated `STRING` column values. ### How was this patch tested? - New cases in `CharVarcharTestSuite` / `BasicCharVarcharTestSuite` / `FileSourceCharVarcharTestSuite` covering `CAST`, least-common-type, transformation, `createDataFrame`, and scan pad/oversize. - Golden file `charvarchar-standard-semantics.sql` (plus analyzer results). - A parity matrix asserting the fixed-point and single-pass analyzers agree under the flag. - A registry-wide inventory test that fails if any function outside the pass-through allowlist returns `CHAR`/`VARCHAR`. - `RowSetUtilsSuite` for unquoted rendering of `CHAR`, `VARCHAR`, and collated `STRING`; verified it fails when the `RowSetUtils` fix is reverted. Closes#58033 from srielau/SPARK-58794. Authored-by: Serge Rielau <serge@rielau.com> Signed-off-by: Gengliang Wang <gengliang@apache.org> (cherry picked from commit 763b7c9) Signed-off-by: Gengliang Wang <gengliang@apache.org>

gengliangwang · 2026-08-19T17:29:43Z

Merge Summary:

merged into master 763b7c9
merged into branch-4.x 79f76f3

Posted by merge_spark_pr.py

…esult schemas ### What changes were proposed in this pull request? Covers Connect JDBC mapping ([SPARK-58806](https://issues.apache.org/jira/browse/SPARK-58806)), plus HiveServer2 / JDBC result metadata ([SPARK-58804](https://issues.apache.org/jira/browse/SPARK-58804)) and `DatabaseMetaData.getColumns` `COLUMN_SIZE` ([SPARK-58805](https://issues.apache.org/jira/browse/SPARK-58805)). Parent: [SPARK-58794](https://issues.apache.org/jira/browse/SPARK-58794). Independent of the remaining CHAR/VARCHAR follow-ups. Rebased onto `master` after #58033; can merge without #58080 / #58087 / #58130. Unique vs master: srielau/spark@master...serge-rielau_data/SPARK-58794-clients Expose first-class `CHAR(n)` and `VARCHAR(n)` to SQL clients when `spark.sql.charVarchar.standardSemantics.enabled` is on, and let those clients decode result rows that carry the types. Metadata: - Spark Connect JDBC maps `CharType` / `VarcharType` to `java.sql.Types.CHAR` / `VARCHAR`, with Java string values and precision / display size `n`. - Connect `DatabaseMetaData.getColumns` reports `COLUMN_SIZE` as the declared character length `n`. `CHAR_OCTET_LENGTH` is the UTF-8 maximum byte capacity `4 * n` (saturating at `Int.MaxValue`); unbounded STRING stays 0. - Connect `DatabaseMetaData.getTypeInfo` lists `CHAR` and `VARCHAR` with `CREATE_PARAMS = length`. Unbounded `STRING` remains a `Types.VARCHAR` row without create params. - HiveServer2 `getColumns` reports `COLUMN_SIZE = n` for `VARCHAR(n)` as well as `CHAR(n)`, and `CHAR_OCTET_LENGTH = 4 * n` for both (null for unbounded STRING). - HiveServer2 result metadata coverage pins Hive JDBC names (`char` / `varchar`, no length) and precision `n` for CAST results. Decode (JVM Connect client): - The Connect client process has no engine `SQLConf`, so `RowEncoder.encoderFor` still follows the local `charVarcharFirstClassTypes` flag and rejects CHAR/VARCHAR when first-class types are off. - `RowEncoder.encoderForResultSchema` always accepts CHAR/VARCHAR. `SparkResult.createEncoder` uses it only for `UnboundRowEncoder` (engine-produced result schemas). Other encoder bindings keep `encoderFor`. - Arrow serializer / deserializer treat `CharEncoder` / `VarcharEncoder` like `StringEncoder`, and deserialize via `StringHelper.plainStringType` so STRING Arrow vectors up-cast to CHAR/VARCHAR. - JDBC CAST `ResultSet.next` / `collect` needs this path; metadata-only mapping is not enough. Python Connect already maps proto `char` / `var_char` in `pyspark.sql.connect.types`; a mixed classic vs Connect test covers schema and `collect()`. The existing Connect proto already carries CHAR/VARCHAR and their lengths. ### Why are the changes needed? Spark Connect JDBC rejected first-class `CharType` and `VarcharType` as unsupported because its metadata mapping only recognized the `StringType` singleton. `getColumns` hardcoded `CHAR_OCTET_LENGTH` to 0, and `getTypeInfo` listed only unbounded STRING. HiveServer2 already identified VARCHAR catalog columns but reported `COLUMN_SIZE` as unknown. Clients therefore cannot reliably describe or size CHAR/VARCHAR columns even though Catalyst retains the type and length. Separately, a server with standard semantics sends CHAR/VARCHAR in the result schema. The JVM Connect client then failed at row decode (`UNSUPPORTED_DATA_TYPE_FOR_ENCODER`) because `RowEncoder.encoderFor` read the client's default conf (`charVarcharFirstClassTypes = false`). JDBC CAST collect and Spark Connect `SparkResult` both hit that path. ### Does this PR introduce _any_ user-facing change? Yes, when first-class CHAR/VARCHAR types are present: JDBC and HiveServer2 metadata now report the corresponding JDBC type and declared character length instead of rejecting the type or reporting an unknown size. `CHAR_OCTET_LENGTH` is the UTF-8 maximum byte capacity (`4 * n`), not the character length. Connect clients can also collect CHAR/VARCHAR result columns instead of failing to decode the schema. ### How was this patch tested? - `RowEncoderSuite`: `encoderFor` still raises `UNSUPPORTED_DATA_TYPE_FOR_ENCODER` for CHAR/VARCHAR when both first-class flags are off; `encoderForResultSchema` accepts the same schema. - `ArrowEncoderSuite`: CHAR/VARCHAR Arrow round-trip (top-level, nested struct, array). - `SparkConnectJdbcDataTypeSuite`: CAST result metadata and table `getColumns` (`CHAR(4)` / `VARCHAR(6)`, `COLUMN_SIZE`, `CHAR_OCTET_LENGTH` = 16 / 24); CAST collect after decode. - `SparkConnectDatabaseMetaDataSuite`: `getTypeInfo` rows for CHAR / VARCHAR. - `SparkMetadataOperationSuite`: HiveServer2 `getColumns` / CAST result metadata (`CHAR_OCTET_LENGTH` = `4 * n`). - `pyspark.sql.tests.connect.test_connect_basic.SparkConnectBasicTests.test_char_varchar_result_schema`: classic vs Connect schema and `collect()`. Local compile of `sql-api`, `connect-client-jdbc`, and `hive-thriftserver` passed. HiveServer2 / full Connect JDBC in this environment remain blocked (`CheckReturnValue` while compiling `connect-common`; FIPS Python multiprocessing). Please treat the suites above as the merge gate. ### Was this patch authored or co-authored using generative AI tooling? Yes (Cursor). Closes#58132 from srielau/serge-rielau_data/SPARK-58794-clients. Authored-by: Serge Rielau <serge@rielau.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

…esult schemas ### What changes were proposed in this pull request? Covers Connect JDBC mapping ([SPARK-58806](https://issues.apache.org/jira/browse/SPARK-58806)), plus HiveServer2 / JDBC result metadata ([SPARK-58804](https://issues.apache.org/jira/browse/SPARK-58804)) and `DatabaseMetaData.getColumns` `COLUMN_SIZE` ([SPARK-58805](https://issues.apache.org/jira/browse/SPARK-58805)). Parent: [SPARK-58794](https://issues.apache.org/jira/browse/SPARK-58794). Independent of the remaining CHAR/VARCHAR follow-ups. Rebased onto `master` after #58033; can merge without #58080 / #58087 / #58130. Unique vs master: srielau/spark@master...serge-rielau_data/SPARK-58794-clients Expose first-class `CHAR(n)` and `VARCHAR(n)` to SQL clients when `spark.sql.charVarchar.standardSemantics.enabled` is on, and let those clients decode result rows that carry the types. Metadata: - Spark Connect JDBC maps `CharType` / `VarcharType` to `java.sql.Types.CHAR` / `VARCHAR`, with Java string values and precision / display size `n`. - Connect `DatabaseMetaData.getColumns` reports `COLUMN_SIZE` as the declared character length `n`. `CHAR_OCTET_LENGTH` is the UTF-8 maximum byte capacity `4 * n` (saturating at `Int.MaxValue`); unbounded STRING stays 0. - Connect `DatabaseMetaData.getTypeInfo` lists `CHAR` and `VARCHAR` with `CREATE_PARAMS = length`. Unbounded `STRING` remains a `Types.VARCHAR` row without create params. - HiveServer2 `getColumns` reports `COLUMN_SIZE = n` for `VARCHAR(n)` as well as `CHAR(n)`, and `CHAR_OCTET_LENGTH = 4 * n` for both (null for unbounded STRING). - HiveServer2 result metadata coverage pins Hive JDBC names (`char` / `varchar`, no length) and precision `n` for CAST results. Decode (JVM Connect client): - The Connect client process has no engine `SQLConf`, so `RowEncoder.encoderFor` still follows the local `charVarcharFirstClassTypes` flag and rejects CHAR/VARCHAR when first-class types are off. - `RowEncoder.encoderForResultSchema` always accepts CHAR/VARCHAR. `SparkResult.createEncoder` uses it only for `UnboundRowEncoder` (engine-produced result schemas). Other encoder bindings keep `encoderFor`. - Arrow serializer / deserializer treat `CharEncoder` / `VarcharEncoder` like `StringEncoder`, and deserialize via `StringHelper.plainStringType` so STRING Arrow vectors up-cast to CHAR/VARCHAR. - JDBC CAST `ResultSet.next` / `collect` needs this path; metadata-only mapping is not enough. Python Connect already maps proto `char` / `var_char` in `pyspark.sql.connect.types`; a mixed classic vs Connect test covers schema and `collect()`. The existing Connect proto already carries CHAR/VARCHAR and their lengths. ### Why are the changes needed? Spark Connect JDBC rejected first-class `CharType` and `VarcharType` as unsupported because its metadata mapping only recognized the `StringType` singleton. `getColumns` hardcoded `CHAR_OCTET_LENGTH` to 0, and `getTypeInfo` listed only unbounded STRING. HiveServer2 already identified VARCHAR catalog columns but reported `COLUMN_SIZE` as unknown. Clients therefore cannot reliably describe or size CHAR/VARCHAR columns even though Catalyst retains the type and length. Separately, a server with standard semantics sends CHAR/VARCHAR in the result schema. The JVM Connect client then failed at row decode (`UNSUPPORTED_DATA_TYPE_FOR_ENCODER`) because `RowEncoder.encoderFor` read the client's default conf (`charVarcharFirstClassTypes = false`). JDBC CAST collect and Spark Connect `SparkResult` both hit that path. ### Does this PR introduce _any_ user-facing change? Yes, when first-class CHAR/VARCHAR types are present: JDBC and HiveServer2 metadata now report the corresponding JDBC type and declared character length instead of rejecting the type or reporting an unknown size. `CHAR_OCTET_LENGTH` is the UTF-8 maximum byte capacity (`4 * n`), not the character length. Connect clients can also collect CHAR/VARCHAR result columns instead of failing to decode the schema. ### How was this patch tested? - `RowEncoderSuite`: `encoderFor` still raises `UNSUPPORTED_DATA_TYPE_FOR_ENCODER` for CHAR/VARCHAR when both first-class flags are off; `encoderForResultSchema` accepts the same schema. - `ArrowEncoderSuite`: CHAR/VARCHAR Arrow round-trip (top-level, nested struct, array). - `SparkConnectJdbcDataTypeSuite`: CAST result metadata and table `getColumns` (`CHAR(4)` / `VARCHAR(6)`, `COLUMN_SIZE`, `CHAR_OCTET_LENGTH` = 16 / 24); CAST collect after decode. - `SparkConnectDatabaseMetaDataSuite`: `getTypeInfo` rows for CHAR / VARCHAR. - `SparkMetadataOperationSuite`: HiveServer2 `getColumns` / CAST result metadata (`CHAR_OCTET_LENGTH` = `4 * n`). - `pyspark.sql.tests.connect.test_connect_basic.SparkConnectBasicTests.test_char_varchar_result_schema`: classic vs Connect schema and `collect()`. Local compile of `sql-api`, `connect-client-jdbc`, and `hive-thriftserver` passed. HiveServer2 / full Connect JDBC in this environment remain blocked (`CheckReturnValue` while compiling `connect-common`; FIPS Python multiprocessing). Please treat the suites above as the merge gate. ### Was this patch authored or co-authored using generative AI tooling? Yes (Cursor). Closes#58132 from srielau/serge-rielau_data/SPARK-58794-clients. Authored-by: Serge Rielau <serge@rielau.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 6af294e) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

### What changes were proposed in this pull request? Covers the string-family LCT lattice ([SPARK-58798](https://issues.apache.org/jira/browse/SPARK-58798)). Parent: [SPARK-58794](https://issues.apache.org/jira/browse/SPARK-58794). Also includes explicit character CAST truncation ([SPARK-58797](https://issues.apache.org/jira/browse/SPARK-58797)). Rebased onto `master` after #58033 merged. Unique commits vs master: srielau/spark@master...SPARK-58794-lct-compare Follow-up that drills into least-common-type, comparison, and `IN` under `spark.sql.charVarchar.standardSemantics.enabled`. `CollationTypeCoercion` is not gated on `standardSemantics` alone (`charVarcharFirstClassTypes` is also true under `preserveCharVarcharTypeInfo`). The equal-collation length LCT therefore applies in preserve-only mode as well; tests cover both flags. **Bug fix.** `CollationTypeCoercion.getWinningStringType` required `sameType` at equal collation strength. Same-collation `CHAR(2)` and `CHAR(4)` therefore looked like a collation mismatch and became `IndeterminateStringType` (`string collate null`). At equal strength, take `StringHelper.tightestCommonString` instead so every child is cast to `max(n, m)` (padding, never truncating). **Explicit CAST vs store assignment (ISO 6.13 / 9.2).** Under the flag: - Character-to-`CHAR`/`VARCHAR` `CAST` / `TRY_CAST` **truncates** to `n` characters (then `CHAR` pads). Example: `CAST('abcdef' AS VARCHAR(2))` is `'ab'`, not `EXCEED_LIMIT_LENGTH`. - Numeric (and other non-string) `CAST` to `CHAR`/`VARCHAR` still **errors** when the formatted literal does not fit (`CAST(12345 AS VARCHAR(4))` -> `EXCEED_LIMIT_LENGTH` / 54006). That matches ISO 22001, not silent digit clipping. - Store assignment (`INSERT`, `Dataset.to`, encoder write-side) is unchanged: non-space overflow still errors. Project store-assignment casts to unconstrained `STRING` then `stringLengthCheck`, so they do not inherit CAST truncation. **Coverage / pinned behavior.** - Set ops: `UNION` / `UNION ALL` / `INTERSECT` / `EXCEPT`, plus multi-row `VALUES`, all share the string-family LCT. - Comparison and `IN`: every participant is cast to that LCT, including the `IN` left-hand side (`InTypeCoercion` uses `findWiderCommonType` over `value +: list`). Casting to `CHAR` pads, so unequal `CHAR` lengths compare equal after widen; casting to `VARCHAR`/`STRING` keeps the `CHAR` pad, so `CHAR 'a'` (stored as `'a '`) is not equal to `VARCHAR`/`STRING 'a'` unless the other side carries the same trailing blank. Ignoring trailing blanks is a collation concern (`RTRIM`), not a type-level `PAD SPACE` policy (design D17). - Parameterized lengths (`CHAR(:n)` / `VARCHAR(?)`): markers in length position already bind through the parser; under the flag, character CAST truncates to the bound `n`, numeric CAST still errors if it does not fit, and DDL keeps the bound `n`. Negative / non-integral lengths fail at parse after substitution. ### Why are the changes needed? #58033 left the collated mixed-length LCT gap as a known issue, and had only thin coverage of `UNION ALL` / `INTERSECT` with no compare / `IN`-LHS / `VALUES` / `EXCEPT` matrix. Without pinning the LCT-cast rule, CHAR vs CHAR vs VARCHAR equality is easy to misread as PAD SPACE. Explicit character CAST was still using the write-side length check, so `CAST('abcdef' AS VARCHAR(2))` failed the same way as `INSERT`. ISO 6.13 truncates that CAST; 9.2 keeps the INSERT error. Numeric CAST must still fail when the literal does not fit (ISO 22001). ### Does this PR introduce _any_ user-facing change? Yes, when the flag is on: - Collated `CHAR`/`VARCHAR` values of different lengths but the same collation now widen to `CHAR`/`VARCHAR(max(n,m))` instead of an indeterminate collation. Compare / `IN` / set-op results for mixed `CHAR` lengths follow the LCT-cast rule above (this matches what the engine already did for non-collated paths; the new tests lock it in). - Explicit character-to-`CHAR`/`VARCHAR` CAST truncates instead of raising `EXCEED_LIMIT_LENGTH`. `INSERT` / encoder overflow and numeric CAST overflow still error. ### How was this patch tested? - New `BasicCharVarcharTestSuite` cases covering compare, `IN` (including analyzed-plan assertion that the LHS widens), `RTRIM`, collated LCT, `UNION` / `INTERSECT` / `EXCEPT`, multi-row `VALUES`, parameterized `CHAR`/`VARCHAR` lengths, character CAST truncation, and numeric CAST overflow under the flag. - Expanded golden file `charvarchar-standard-semantics.sql`. - Locally: `BasicCharVarcharTestSuite` and the golden file regeneration/run. Closes#58080 from srielau/SPARK-58794-lct-compare. Authored-by: Serge Rielau <serge@rielau.com> Signed-off-by: Yicong-Huang <17627829+Yicong-Huang@users.noreply.github.com>

### What changes were proposed in this pull request? Covers the string-family LCT lattice ([SPARK-58798](https://issues.apache.org/jira/browse/SPARK-58798)). Parent: [SPARK-58794](https://issues.apache.org/jira/browse/SPARK-58794). Also includes explicit character CAST truncation ([SPARK-58797](https://issues.apache.org/jira/browse/SPARK-58797)). Rebased onto `master` after #58033 merged. Unique commits vs master: srielau/spark@master...SPARK-58794-lct-compare Follow-up that drills into least-common-type, comparison, and `IN` under `spark.sql.charVarchar.standardSemantics.enabled`. `CollationTypeCoercion` is not gated on `standardSemantics` alone (`charVarcharFirstClassTypes` is also true under `preserveCharVarcharTypeInfo`). The equal-collation length LCT therefore applies in preserve-only mode as well; tests cover both flags. **Bug fix.** `CollationTypeCoercion.getWinningStringType` required `sameType` at equal collation strength. Same-collation `CHAR(2)` and `CHAR(4)` therefore looked like a collation mismatch and became `IndeterminateStringType` (`string collate null`). At equal strength, take `StringHelper.tightestCommonString` instead so every child is cast to `max(n, m)` (padding, never truncating). **Explicit CAST vs store assignment (ISO 6.13 / 9.2).** Under the flag: - Character-to-`CHAR`/`VARCHAR` `CAST` / `TRY_CAST` **truncates** to `n` characters (then `CHAR` pads). Example: `CAST('abcdef' AS VARCHAR(2))` is `'ab'`, not `EXCEED_LIMIT_LENGTH`. - Numeric (and other non-string) `CAST` to `CHAR`/`VARCHAR` still **errors** when the formatted literal does not fit (`CAST(12345 AS VARCHAR(4))` -> `EXCEED_LIMIT_LENGTH` / 54006). That matches ISO 22001, not silent digit clipping. - Store assignment (`INSERT`, `Dataset.to`, encoder write-side) is unchanged: non-space overflow still errors. Project store-assignment casts to unconstrained `STRING` then `stringLengthCheck`, so they do not inherit CAST truncation. **Coverage / pinned behavior.** - Set ops: `UNION` / `UNION ALL` / `INTERSECT` / `EXCEPT`, plus multi-row `VALUES`, all share the string-family LCT. - Comparison and `IN`: every participant is cast to that LCT, including the `IN` left-hand side (`InTypeCoercion` uses `findWiderCommonType` over `value +: list`). Casting to `CHAR` pads, so unequal `CHAR` lengths compare equal after widen; casting to `VARCHAR`/`STRING` keeps the `CHAR` pad, so `CHAR 'a'` (stored as `'a '`) is not equal to `VARCHAR`/`STRING 'a'` unless the other side carries the same trailing blank. Ignoring trailing blanks is a collation concern (`RTRIM`), not a type-level `PAD SPACE` policy (design D17). - Parameterized lengths (`CHAR(:n)` / `VARCHAR(?)`): markers in length position already bind through the parser; under the flag, character CAST truncates to the bound `n`, numeric CAST still errors if it does not fit, and DDL keeps the bound `n`. Negative / non-integral lengths fail at parse after substitution. ### Why are the changes needed? #58033 left the collated mixed-length LCT gap as a known issue, and had only thin coverage of `UNION ALL` / `INTERSECT` with no compare / `IN`-LHS / `VALUES` / `EXCEPT` matrix. Without pinning the LCT-cast rule, CHAR vs CHAR vs VARCHAR equality is easy to misread as PAD SPACE. Explicit character CAST was still using the write-side length check, so `CAST('abcdef' AS VARCHAR(2))` failed the same way as `INSERT`. ISO 6.13 truncates that CAST; 9.2 keeps the INSERT error. Numeric CAST must still fail when the literal does not fit (ISO 22001). ### Does this PR introduce _any_ user-facing change? Yes, when the flag is on: - Collated `CHAR`/`VARCHAR` values of different lengths but the same collation now widen to `CHAR`/`VARCHAR(max(n,m))` instead of an indeterminate collation. Compare / `IN` / set-op results for mixed `CHAR` lengths follow the LCT-cast rule above (this matches what the engine already did for non-collated paths; the new tests lock it in). - Explicit character-to-`CHAR`/`VARCHAR` CAST truncates instead of raising `EXCEED_LIMIT_LENGTH`. `INSERT` / encoder overflow and numeric CAST overflow still error. ### How was this patch tested? - New `BasicCharVarcharTestSuite` cases covering compare, `IN` (including analyzed-plan assertion that the LHS widens), `RTRIM`, collated LCT, `UNION` / `INTERSECT` / `EXCEPT`, multi-row `VALUES`, parameterized `CHAR`/`VARCHAR` lengths, character CAST truncation, and numeric CAST overflow under the flag. - Expanded golden file `charvarchar-standard-semantics.sql`. - Locally: `BasicCharVarcharTestSuite` and the golden file regeneration/run. Closes#58080 from srielau/SPARK-58794-lct-compare. Authored-by: Serge Rielau <serge@rielau.com> Signed-off-by: Yicong-Huang <17627829+Yicong-Huang@users.noreply.github.com> (cherry picked from commit ac5bba0) Signed-off-by: Yicong-Huang <17627829+Yicong-Huang@users.noreply.github.com>

srielau added 5 commits August 16, 2026 18:24

[SPARK-58794][SQL] Add standardSemantics foundation for CHAR/VARCHAR
b67ae50
Introduce spark.sql.charVarchar.standardSemantics.enabled and wire first-class types, real CAST, LCT, R1 STRING results, write-side typing, and read-side pad/oversize checks (SPARK-58796 through SPARK-58801).

[SPARK-58794][SQL] Apply scalafmt to sql/api standardSemantics changes
9b32c73

cloud-fan reviewed Aug 17, 2026
View reviewed changes

[SPARK-58794][SQL] Retrigger CI
7601420

srielau mentioned this pull request Aug 18, 2026
[SPARK-58802][SQL] Dual-run coverage for CHAR/VARCHAR standardSemantics #58057
Closed

srielau mentioned this pull request Aug 18, 2026
[SPARK-58803][SQL] Dataset/encoder/UDF CHAR/VARCHAR under standardSemantics #58059
Closed

srielau added 2 commits August 18, 2026 01:58

srielau commented Aug 18, 2026
View reviewed changes

srielau mentioned this pull request Aug 18, 2026
[SPARK-58798][SQL] Fix collated CHAR LCT and cover compare/IN/set-ops #58080
Closed

cloud-fan approved these changes Aug 18, 2026
View reviewed changes

Comment threadsql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala Outdated

srielau added 2 commits August 18, 2026 16:24

[SPARK-58794][SQL] Narrow inventory test claim to exercised argument …
9f4d006
…shapes

[SPARK-58794][SQL] Replace non-ASCII dash in inventory test comment
8b12f71

This was referenced Aug 18, 2026
[SPARK-58807][SQL] Preserve CHAR/VARCHAR on language surfaces under standardSemantics #58087
Closed
[SPARK-58794][SQL] Promote CHAR/VARCHAR to STRING via ImplicitTypeCasts #58130
Closed

gengliangwang approved these changes Aug 19, 2026
View reviewed changes

gengliangwang closed this in 763b7c9Aug 19, 2026

srielau mentioned this pull request Aug 19, 2026
[SPARK-58806][SQL][CONNECT] Expose CHAR/VARCHAR metadata and decode result schemas #58132
Closed

Navigation Menu

Uh oh!

[SPARK-58794][SQL] Add standardSemantics foundation for CHAR/VARCHAR - #58033

[SPARK-58794][SQL] Add standardSemantics foundation for CHAR/VARCHAR#58033
srielau wants to merge 13 commits into
apache:masterfrom
srielau:SPARK-58794

srielau commented Aug 16, 2026•
edited
Loading

Uh oh!

cloud-fan left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

srielau commented Aug 17, 2026

Uh oh!

srielau left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cloud-fan left a comment

Uh oh!

Uh oh!

gengliangwang commented Aug 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

srielau commented Aug 16, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Out of scope / follow-up

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

cloud-fan left a comment

Choose a reason for hiding this comment

Correctness (1)

Suggestions (1)

Verification

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

srielau commented Aug 17, 2026

Uh oh!

srielau left a comment

Choose a reason for hiding this comment

SQL Language review

Findings not attachable to the diff

Recommended actions

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cloud-fan left a comment

Choose a reason for hiding this comment

Verification

Uh oh!

Uh oh!

gengliangwang commented Aug 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

srielau commented Aug 16, 2026•
edited
Loading