Uh oh!
There was an error while loading. Please reload this page.
[SPARK-33709][SQL] Refactor FileTable - #30670
Conversation
| } | ||
| lazy val dataSchema: StructType = { | ||
| StructType(fileSchema.filterNot(partitionSchema.contains)) |
There was a problem hiding this comment.
should we consider caseSensitive option as line 106 ~ line 109:
val fields = fileSchema.fields.filterNot { field =>
val colName = PartitioningUtils.getColName(field, caseSensitive)
partitionNameSet.contains(colName)
}
There was a problem hiding this comment.
Yes. Let's see if there are other failures.
There was a problem hiding this comment.
3 test in HiveThriftHttpServerSuite failed, seems these three use cases fail in other pr too ....
SparkQA
commented
Dec 8, 2020
Kubernetes integration test starting |
SparkQA
commented
Dec 8, 2020
Kubernetes integration test status failure |
SparkQA
commented
Dec 8, 2020
Test build #132421 has finished for PR 30670 at commit
|
| val isCaseSensitive = sqlConf.caseSensitiveAnalysis | ||
| val parquetSchema = | ||
| new SparkToParquetSchemaConverter(sparkSession.sessionState.conf).convert(schema) | ||
| new SparkToParquetSchemaConverter(sparkSession.sessionState.conf).convert(dataSchema) |
There was a problem hiding this comment.
Maybe we should use readDataSchema() instead of dataSchema
What changes were proposed in this pull request?
Why are the changes needed?
Does this PR introduce any user-facing change?
How was this patch tested?