Uh oh!
There was an error while loading. Please reload this page.
[SPARK-19833][SQL]remove SQLConf.HIVE_VERIFY_PARTITION_PATH, always return empty when the location does not exists - #17176
Conversation
…eturn empty when the location does not exists
SparkQA
commented
Mar 6, 2017
Test build #73991 has finished for PR 17176 at commit
|
4bb0e28 to
8128567Compare| } | ||
| // convert /demo/data/year/month/day to /demo/data/*/*/*/ | ||
| def getPathPatternByPath(parNum: Int, tempPath: Path, partitionName: String): String = { | ||
| // if the partition path does not end with partition name, we should not |
There was a problem hiding this comment.
if the partition location has been altered to another location, we should not do this pattern, or we will list pattern files which does not belong to the partition
SparkQA
commented
Mar 6, 2017
Test build #73998 has finished for PR 17176 at commit
|
SparkQA
commented
Mar 6, 2017
Test build #73992 has finished for PR 17176 at commit
|
windpiger
commented
Mar 6, 2017
retest this please |
SparkQA
commented
Mar 6, 2017
Test build #74016 has finished for PR 17176 at commit
|
windpiger
commented
Mar 7, 2017
why jenkins failed... |
SparkQA
commented
Mar 7, 2017
Test build #74072 has finished for PR 17176 at commit
|
| def verifyPartitionPath( | ||
| partitionToDeserializer: Map[HivePartition, Class[_ <: Deserializer]]): | ||
| Map[HivePartition, Class[_ <: Deserializer]] = { | ||
| if (!sparkSession.sessionState.conf.verifyPartitionPath) { |
There was a problem hiding this comment.
after this pr https://github.com/apache/spark/pull/17187， read hive table which does not use stored by will not use HiveTableScanExec.
this function has a bug ,that if the partition path is custom path
- it will still do filter for all partition path in the parameter
partitionToDeserializer, - it will scan the path which does not belong to the table ,e.g. custom path is
/root/a
and the partitionSpec isb=1/c=2, this will lead to scan/because of thegetPathPatternByPath
SparkQA
commented
Mar 7, 2017
Test build #74106 has finished for PR 17176 at commit
|
SparkQA
commented
Mar 7, 2017
Test build #74107 has finished for PR 17176 at commit
|
gatorsmile
commented
Oct 28, 2017
@windpiger If you do not have a bandwidth to continue it, how about closing it now? |
| case (partition, partDeserializer) => | ||
| val partPath = partition.getDataLocation | ||
| val fs = partPath.getFileSystem(hadoopConf) | ||
| fs.exists(partPath) |
There was a problem hiding this comment.
Each partition sending an RPC request to the NameNode can result in poor performance
Closesapache#21766Closesapache#21679Closesapache#21161Closesapache#20846Closesapache#19434Closesapache#18080Closesapache#17648Closesapache#17169 Add: Closesapache#22813Closesapache#21994Closesapache#22005Closesapache#22463 Add: Closesapache#15899 Add: Closesapache#22539Closesapache#21868Closesapache#21514Closesapache#21402Closesapache#21322Closesapache#21257Closesapache#20163Closesapache#19691Closesapache#18697Closesapache#18636Closesapache#17176Closesapache#23001 from wangyum/CloseStalePRs. Authored-by: Yuming Wang <yumwang@ebay.com> Signed-off-by: hyukjinkwon <gurwls223@apache.org>
What changes were proposed in this pull request?
In SPARK-5068, we introduce a SQLConf spark.sql.hive.verifyPartitionPath,
if it is set to true, it will avoid the task failed when the patition location does not exists in the filesystem.
this situation should always return emtpy and don't lead to the task failed, here we remove this conf.
And the function
verifyPartitionPathhas a bug ,that if the partition path is custom pathit will still do filter for all partition path in the parameter
partitionToDeserializer,it will scan the path which does not belong to the table ,e.g. custom path is /root/a
and the partitionSpec is b=1/c=2, this will lead to scan / because of the getPathPatternByPath
How was this patch tested?
modify a test case