Uh oh!
There was an error while loading. Please reload this page.
[SPARK-38350][SQL] Make the table name output of V1/V2 "desc extended table" consistent - #35681
[SPARK-38350][SQL] Make the table name output of V1/V2 "desc extended table" consistent#35681gengliangwang wants to merge 1 commit into
Conversation
gengliangwang
commented
Feb 28, 2022
| rows += toCatalystRow("# Detailed Table Information", "", "") | ||
| rows += toCatalystRow("Name", table.name(), "") | ||
| rows += toCatalystRow("Catalog", resolvedTable.catalog.name(), "") | ||
| rows += toCatalystRow("Database", resolvedTable.identifier.namespace().mkString("."), "") |
There was a problem hiding this comment.
is namespace() returning a list? Should this be a single identifier?
There was a problem hiding this comment.
Yes, namespace() returns a list. It's a feature of DS V2. Usually the length should be 1.
There was a problem hiding this comment.
so if it returns a list will the Database name contain periods?
There was a problem hiding this comment.
If the length of the namespace is 1, there is no any period
There was a problem hiding this comment.
we spoke online with Wenchen and Gengliang, didn't realize the namespace is a multi part name.
| " COMMENT 'this is a test table'" + | ||
| " LOCATION 'file:/tmp/testcat/table_name'") | ||
| val descriptionDf = spark.sql("DESCRIBE TABLE EXTENDED testcat.table_name") | ||
| val descriptionDf = spark.sql("DESCRIBE TABLE EXTENDED testcat.default.table_name") |
There was a problem hiding this comment.
Does this also fix SHOW TABLE EXTENDED?
There was a problem hiding this comment.
There is no V2 version of SHOW TABLE EXTENDED yet.
What changes were proposed in this pull request?
The V1 "DESC TABLE EXTENDED" command contains info of "Database" and "Table", which can be used by external tools like DBT: #17394
However, the V2 version contains only one field "name" representing "catalog.database.table". External tools can't recognize it.
Also, it is weird to have different command outputs from the same command.
This PR is to make the database/table name output of V1/V2 "desc extended table" consistent. For the output of V2 tables, this PR also shows the catalog name in the output.
Why are the changes needed?
Make the database/table name output of V1/V2 "desc extended table" consistent so that it can be parsed by external tools like DBT.
Does this PR introduce any user-facing change?
Yes, it changes the database/table name output of V2 "desc extended table" so that it becomes consistent with V1 tables.
How was this patch tested?
UT