[SPARK-58806][SQL][CONNECT] Expose CHAR/VARCHAR metadata and decode result schemas

[srielau]

What changes were proposed in this pull request?

Covers Connect JDBC mapping (SPARK-58806), plus HiveServer2 / JDBC result metadata (SPARK-58804) and DatabaseMetaData.getColumnsCOLUMN_SIZE (SPARK-58805). Parent: SPARK-58794.

Independent of the remaining CHAR/VARCHAR follow-ups. Rebased onto master after #58033; can merge without #58080 / #58087 / #58130. Unique vs master: srielau/spark@master...serge-rielau_data/SPARK-58794-clients

Expose first-class CHAR(n) and VARCHAR(n) to SQL clients when spark.sql.charVarchar.standardSemantics.enabled is on, and let those clients decode result rows that carry the types.

Metadata:

• Spark Connect JDBC maps CharType / VarcharType to java.sql.Types.CHAR / VARCHAR, with Java string values and precision / display size n.
• Connect DatabaseMetaData.getColumns reports COLUMN_SIZE as the declared character length n. CHAR_OCTET_LENGTH is the UTF-8 maximum byte capacity 4 * n (saturating at Int.MaxValue); unbounded STRING stays 0.
• Connect DatabaseMetaData.getTypeInfo lists CHAR and VARCHAR with CREATE_PARAMS = length. Unbounded STRING remains a Types.VARCHAR row without create params.
• HiveServer2 getColumns reports COLUMN_SIZE = n for VARCHAR(n) as well as CHAR(n), and CHAR_OCTET_LENGTH = 4 * n for both (null for unbounded STRING).
• HiveServer2 result metadata coverage pins Hive JDBC names (char / varchar, no length) and precision n for CAST results.

Decode (JVM Connect client):

• The Connect client process has no engine SQLConf, so RowEncoder.encoderFor still follows the local charVarcharFirstClassTypes flag and rejects CHAR/VARCHAR when first-class types are off.
• RowEncoder.encoderForResultSchema always accepts CHAR/VARCHAR. SparkResult.createEncoder uses it only for UnboundRowEncoder (engine-produced result schemas). Other encoder bindings keep encoderFor.
• Arrow serializer / deserializer treat CharEncoder / VarcharEncoder like StringEncoder, and deserialize via StringHelper.plainStringType so STRING Arrow vectors up-cast to CHAR/VARCHAR.
• JDBC CAST ResultSet.next / collect needs this path; metadata-only mapping is not enough.

Python Connect already maps proto char / var_char in pyspark.sql.connect.types; a mixed classic vs Connect test covers schema and collect().

The existing Connect proto already carries CHAR/VARCHAR and their lengths.

Why are the changes needed?

Spark Connect JDBC rejected first-class CharType and VarcharType as unsupported because its metadata mapping only recognized the StringType singleton. getColumns hardcoded CHAR_OCTET_LENGTH to 0, and getTypeInfo listed only unbounded STRING. HiveServer2 already identified VARCHAR catalog columns but reported COLUMN_SIZE as unknown. Clients therefore cannot reliably describe or size CHAR/VARCHAR columns even though Catalyst retains the type and length.

Separately, a server with standard semantics sends CHAR/VARCHAR in the result schema. The JVM Connect client then failed at row decode (UNSUPPORTED_DATA_TYPE_FOR_ENCODER) because RowEncoder.encoderFor read the client's default conf (charVarcharFirstClassTypes = false). JDBC CAST collect and Spark Connect SparkResult both hit that path.

Does this PR introduce any user-facing change?

Yes, when first-class CHAR/VARCHAR types are present: JDBC and HiveServer2 metadata now report the corresponding JDBC type and declared character length instead of rejecting the type or reporting an unknown size. CHAR_OCTET_LENGTH is the UTF-8 maximum byte capacity (4 * n), not the character length. Connect clients can also collect CHAR/VARCHAR result columns instead of failing to decode the schema.

How was this patch tested?

• RowEncoderSuite: encoderFor still raises UNSUPPORTED_DATA_TYPE_FOR_ENCODER for CHAR/VARCHAR when both first-class flags are off; encoderForResultSchema accepts the same schema.
• ArrowEncoderSuite: CHAR/VARCHAR Arrow round-trip (top-level, nested struct, array).
• SparkConnectJdbcDataTypeSuite: CAST result metadata and table getColumns (CHAR(4) / VARCHAR(6), COLUMN_SIZE, CHAR_OCTET_LENGTH = 16 / 24); CAST collect after decode.
• SparkConnectDatabaseMetaDataSuite: getTypeInfo rows for CHAR / VARCHAR.
• SparkMetadataOperationSuite: HiveServer2 getColumns / CAST result metadata (CHAR_OCTET_LENGTH = 4 * n).
• pyspark.sql.tests.connect.test_connect_basic.SparkConnectBasicTests.test_char_varchar_result_schema: classic vs Connect schema and collect().

Local compile of sql-api, connect-client-jdbc, and hive-thriftserver passed. HiveServer2 / full Connect JDBC in this environment remain blocked (CheckReturnValue while compiling connect-common; FIPS Python multiprocessing). Please treat the suites above as the merge gate.

Was this patch authored or co-authored using generative AI tooling?

Yes (Cursor).

[srielau]

SQL Language review

Client-metadata review (Connect JDBC + HiveServer2 getColumns). Not Analyzer/Parser.

The Connect JDBC mapping is the real fix: CharType / VarcharType are StringType subclasses, so case StringType => missed them and threw SQLFeatureNotSupportedException. Mapping them to Types.CHAR / VARCHAR with precision n, and widening the other matchers to _: StringType, is the right split.

HS2 getColumns already mapped VARCHAR to Types.VARCHAR; only COLUMN_SIZE was missing. Execute-path CHAR/VARCHAR on HS2 was already wired (TTypeId + CHARACTER_MAXIMUM_LENGTH via defaultSize, which equals length).

Findings

1. Medium -- Connect getColumns still reports CHAR_OCTET_LENGTH = 0. Catalog metadata will pick up COLUMN_SIZE = n via getDisplaySize -> getPrecision, but octet length is hardcoded to 0. HS2 leaves it null as well. Clients that size buffers from CHAR_OCTET_LENGTH still see unknown width.

2. Medium -- getTypeInfo still has no CHAR / VARCHAR(n). Only STRING is listed as Types.VARCHAR. Tools that discover types from getTypeInfo will not see CHAR.

3. Medium -- HS2 getColumns test asserts defaultSize instead of .length. VARCHAR now falls through to colSize === o.defaultSize. That equals length today, but defaultSize is the wrong contract (STRING is 20). This test also does not enable standardSemantics, unlike the new result-metadata test.

4. Low -- HS2 result-metadata test does not exercise this diff. Execute-path CHAR/VARCHAR was already implemented. It pins Hive names char / varchar (no length) vs Connect CHAR(4) / VARCHAR(6), and does not prove the VARCHAR COLUMN_SIZE fix.

5. Medium -- tests were not run. The PR says Connect and HS2 suites were blocked locally. Please run them (or wait on CI) before merge; Hive type-name assertions are easy to get wrong.

6. Low -- two metadata mappers stay in lockstep. Connect getColumns uses display size for COLUMN_SIZE; HS2 uses length. They agree for CHAR/VARCHAR only because display size delegates to precision.

Suggested order

1. Add Connect getColumns coverage for table CHAR/VARCHAR (COLUMN_SIZE, type code, type name).
2. Decide on CHAR_OCTET_LENGTH and getTypeInfo (fix or explicitly out of scope).
3. Assert HS2 VARCHAR size as .length, not defaultSize.
4. Run the two suites before merge.

[srielau]

SQL Language re-review (4673559)

Previous findings are addressed in this commit.

#	Original	Status
1	Connect CHAR_OCTET_LENGTH hardcoded 0; no table getColumns test	Fixed -- JdbcTypeUtils.getCharOctetLength, table getColumns test
2	getTypeInfo missing CHAR / VARCHAR(n)	Fixed -- CHAR + VARCHAR with CREATE_PARAMS = length, ordered by DATA_TYPE then TYPE_NAME
3	HS2 test asserted defaultSize	Fixed -- match .length, explicit 255 / 1024, standardSemantics on
4	HS2 result-metadata test did not prove getColumnSize	Fixed -- size/octet asserted on c17/c18; CAST test kept as Hive name pin
5	Tests not run	Open (environment) -- still blocked locally; relying on CI
6	Dual mappers / display-size vs length	Acceptable -- CHAR/VARCHAR length and octet helpers agree; Connect COLUMN_SIZE still display size, which equals precision for these types

HS2 scaladoc and CHAR_OCTET_LENGTH are updated. Connect table getColumns covers type code, CHAR(4) / VARCHAR(6) names, COLUMN_SIZE, and octet length.

Remaining

1. Medium -- still no local run of the Connect/HS2 suites. The new getColumns / getTypeInfo assertions are the kind that fail on type-name casing. Please confirm GitHub CI for SparkConnectJdbcDataTypeSuite, SparkConnectDatabaseMetaDataSuite, and SparkMetadataOperationSuite is green before merge.

2. Low -- PR description is stale. It still describes only CAST result metadata and "tests blocked", and does not mention CHAR_OCTET_LENGTH or getTypeInfo. Please refresh What/Why/How-tested.

3. Low -- CHAR_OCTET_LENGTH is character length n, not UTF-8 octets. Documented in the helper; JDBC's name says octets. Fine if called out; some clients will allocate n bytes for CHAR(n) and truncate multibyte text.

The production mapping looks correct. Not blocking on (2)/(3).

[srielau]

Re-review remaining items:

1. CI -- not green yet. Fork Build for 4673559 is queued/in progress: https://github.com/srielau/spark/actions/runs/32291730651 (Check changes passed; remaining jobs still queued). The earlier Build on this branch was cancelled by the follow-up push. Local runs of SparkConnectJdbcDataTypeSuite, SparkConnectDatabaseMetaDataSuite, and SparkMetadataOperationSuite are still blocked here. Those three suites remain the merge gate.

2. PR description -- refreshed: CHAR_OCTET_LENGTH, getTypeInfo CHAR/VARCHAR rows, table getColumns, and the How-tested list now match 4673559.

3. Octet length vs UTF-8 bytes -- called out in What/Why/user-facing. Helpers already document that Spark reports character length n (Connect 0 / HS2 null for unbounded STRING). No production change.

[srielau]

SQL Language re-review (b798e6a)

Metadata follow-ups from 4673559 still look in place. The new commit is the right split: engine-produced schemas accept CHAR/VARCHAR (encoderForResultSchema); user-supplied schemas (UDF, Encoders.row, createDataFrame) still honor local charVarcharFirstClassTypes. Arrow treats CHAR/VARCHAR as string vectors and reads via plainStringType, which matches how the server encodes them.

SparkResult only swaps UnboundRowEncoder; typed as[String] keeps StringEncoder and already matched the string-vector case. Nested struct/array CHAR is covered by the recursive flag and the ArrowEncoderSuite round trip.

This decode path is also what makes the existing JDBC CAST getString test runnable on a client with default conf.

Remaining

1. Medium -- PR description is stale. It still talks only about JDBC/HS2 metadata and local test blockage. Please add the Connect result-decode problem (client DefaultSqlApiConf cannot see server standardSemantics) and encoderForResultSchema.

2. Medium -- missing negative test on the user encoder path. Please assert RowEncoder.encoderFor(StructType().add("c", CharType(4))) still throws UNSUPPORTED_DATA_TYPE_FOR_ENCODER when first-class types are off, so the new true override cannot leak into UDF / createDataFrame.

3. Low -- Python Connect is not covered. This is JVM RowEncoder / Arrow. If Python already maps proto CHAR to string, say so in the PR; otherwise it is a follow-up.

4. Low (from last review) -- CHAR_OCTET_LENGTH is character n, not UTF-8 octets; still documented. Confirm CI for JDBC, ArrowEncoderSuite, and HS2.

Architecture of the new entry point looks sound. Not blocking on (3)/(4).

[cloud-fan]

1 blocking, 0 non-blocking, 2 nits.
The result-schema decoding split is sound, but the new octet-length metadata underreports multibyte values and should be corrected before merge.

Correctness (1)

• Blocking: sql/connect/client/jdbc/src/main/scala/org/apache/spark/sql/connect/client/jdbc/util/JdbcTypeUtils.scala:181: Report byte capacity, not character count, in CHAR_OCTET_LENGTH. -- see inline

Nits: 2 minor items (see inline comments).

Verification

I traced engine-produced schemas from SparkResult.createEncoder through RowEncoder.encoderForResultSchema into the Arrow string reader, while the ordinary encoderFor path still consults the local configuration. I also traced both JDBC getColumns implementations: each now returns the declared character count for CHAR_OCTET_LENGTH, even though the added comments identify byte-buffer sizing clients, so a multibyte UTF-8 value can require more bytes than reported.

[cloud-fan]

3 addressed, 0 remaining, 1 new to this AI review. (0 newly introduced, 1 late catch, 0 previously raised.)
0 blocking, 0 non-blocking, 1 nit.
The prior correctness issue is fixed; one small Scaladoc clarity nit remains.

Nits: 1 minor item (see inline comments).

Verification

I traced Connect result schemas from SparkResult.createEncoder through RowEncoder.encoderForResultSchema into Arrow deserialization, confirming that ordinary encoderFor remains configuration-gated. I also traced both JDBC getColumns paths and verified that their current helpers report saturated 4 * n byte capacity, with focused tests asserting the corresponding metadata.

[cloud-fan]

Merge Summary:

• merged into master 6af294e
• merged into branch-4.x f48d84f

Posted by merge_spark_pr.py