Remove redundant table properties when KSHC create table

[maomaodev]

Why are the changes needed?

Fixed#6443. When creating an external table via KSHC with EXTERNAL and LOCATION specified, the KSHC table properties will include external=true(lowercase). Meanwhile, Hive Metastore automatically appends EXTERNAL=TRUE (uppercase). On some metastore backends (e.g. MySQL with a case-insensitive collation), this leads to a duplicate primary key conflict in TABLE_PARAMS.

This PR removes redundant table properties when KSHC creates tables. For example, the external property will be excluded.

How was this patch tested?

UT

Was this patch authored or co-authored using generative AI tooling?

No

[pan3793]

how does this happen for SQL cases? I thought it should be handled by AstBuilder.cleanTableProperties, but it doesn't seem to be? could you please summarize the whole flow of the properties propagation?

[maomaodev]

how does this happen for SQL cases? I thought it should be handled by AstBuilder.cleanTableProperties, but it doesn't seem to be? could you please summarize the whole flow of the properties propagation?

1. SQL case reference: [Bug] Kyuubi Hive Connector Can not Create External Table #6443. The simplest example is:

CREATE EXTERNAL TABLE kyuubi_catalog.database.table_1 (
column1 string
)
LOCATION 'hdfs:///tmp/table_1';

2. In the parse phase, AstBuilder.cleanTableProperties is responsible for filtering/intercepting reserved properties from TBLPROPERTIES. For example, you cannot set reserved properties via TBLPROPERTIES (e.g., trying to use TBLPROPERTIES ('provider'='test') is not allowed).

3. In the physical plan phase, before CreateTableExec calls catalog.createTable(...), it invokes CatalogV2Util.convertTableProperties, which unify all properties (properties, options, serde, location, external, provider, etc.). The resulting map is passed as the properties argument to KSHC’s HiveTableCatalog.createTable. See: [SPARK-31257][SPARK-33561][SQL] Unify create table syntax spark#28026

[pan3793]

thanks, merged to master