Uh oh!
There was an error while loading. Please reload this page.
[SPARK-16408][SQL] SparkSQL Added file get Exception: is a directory … - #14085
[SPARK-16408][SQL] SparkSQL Added file get Exception: is a directory …#14085zenglinxi0615 wants to merge 1 commit into
Conversation
…and recursive is not turned on
AmplabJenkins
commented
Jul 7, 2016
Can one of the admins verify this patch? |
| override def run(sqlContext: SQLContext): Seq[Row] = { | ||
| val hiveContext = sqlContext.asInstanceOf[HiveContext] | ||
| val recursive = sqlContext.sparkContext.getConf.getBoolean("spark.input.dir.recursive", false) |
There was a problem hiding this comment.
I'm not sure these are semantics that are supported by the SQL dialect in Spark SQL. In any event the name of this property is too generic, and I don't think it is something that is set globally.
There was a problem hiding this comment.
I'm pretty sure that it's supported by the SQL dialect in Spark SQL.
And about "the name of this property is too generic, and I don't think it is something that is set globally", do you think we should use another name? and the default value should be true?
There was a problem hiding this comment.
And by the way, I have tried:val recursive = hiveContext.getConf("spark.input.dir.recursive", "false")
but this can only work in spark sql by execute set spark.input.dir.recursive=true before add file, and we can't set the value by --conf spark.input.dir.recursive=true. This makes it difficult for us to move some hive sql directly to SparkSQL.
There was a problem hiding this comment.
Adding this session-scoped configuration is risky. If needed, we can improve the SQL syntax for supporting it.
There was a problem hiding this comment.
I was wondering if we could call:
sparkSession.sparkContext.addFile(path, true)
in AddFileCommand func, since it's a general demand in ETL.
jinxing64
commented
Jun 20, 2017
@zenglinxi0615 Just one quick question, could we give a default configuration for |
HyukjinKwon
commented
Jul 24, 2017
@zenglinxi0615 Could you answer to the question above if you are active? |
## What changes were proposed in this pull request? This PR proposes to close stale PRs, mostly the same instances with apache#18017Closesapache#14085 - [SPARK-16408][SQL] SparkSQL Added file get Exception: is a directory … Closesapache#14239 - [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism to accelerate shuffle stage. Closesapache#14567 - [SPARK-16992][PYSPARK] Python Pep8 formatting and import reorganisation Closesapache#14579 - [SPARK-16921][PYSPARK] RDD/DataFrame persist()/cache() should return Python context managers Closesapache#14601 - [SPARK-13979][Core] Killed executor is re spawned without AWS key… Closesapache#14830 - [SPARK-16992][PYSPARK][DOCS] import sort and autopep8 on Pyspark examples Closesapache#14963 - [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in lint-python Closesapache#15227 - [SPARK-17655][SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage Closesapache#15240 - [SPARK-17556] [CORE] [SQL] Executor side broadcast for broadcast joins Closesapache#15405 - [SPARK-15917][CORE] Added support for number of executors in Standalone [WIP] Closesapache#16099 - [SPARK-18665][SQL] set statement state to "ERROR" after user cancel job Closesapache#16445 - [SPARK-19043][SQL]Make SparkSQLSessionManager more configurable Closesapache#16618 - [SPARK-14409][ML][WIP] Add RankingEvaluator Closesapache#16766 - [SPARK-19426][SQL] Custom coalesce for Dataset Closesapache#16832 - [SPARK-19490][SQL] ignore case sensitivity when filtering hive partition columns Closesapache#17052 - [SPARK-19690][SS] Join a streaming DataFrame with a batch DataFrame which has an aggregation may not work Closesapache#17267 - [SPARK-19926][PYSPARK] Make pyspark exception more user-friendly Closesapache#17371 - [SPARK-19903][PYSPARK][SS] window operator miss the `watermark` metadata of time column Closesapache#17401 - [SPARK-18364][YARN] Expose metrics for YarnShuffleService Closesapache#17519 - [SPARK-15352][Doc] follow-up: add configuration docs for topology-aware block replication Closesapache#17530 - [SPARK-5158] Access kerberized HDFS from Spark standalone Closesapache#17854 - [SPARK-20564][Deploy] Reduce massive executor failures when executor count is large (>2000) Closesapache#17979 - [SPARK-19320][MESOS][WIP]allow specifying a hard limit on number of gpus required in each spark executor when running on mesos Closesapache#18127 - [SPARK-6628][SQL][Branch-2.1] Fix ClassCastException when executing sql statement 'insert into' on hbase table Closesapache#18236 - [SPARK-21015] Check field name is not null and empty in GenericRowWit… Closesapache#18269 - [SPARK-21056][SQL] Use at most one spark job to list files in InMemoryFileIndex Closesapache#18328 - [SPARK-21121][SQL] Support changing storage level via the spark.sql.inMemoryColumnarStorage.level variable Closesapache#18354 - [SPARK-18016][SQL][CATALYST][BRANCH-2.1] Code Generation: Constant Pool Limit - Class Splitting Closesapache#18383 - [SPARK-21167][SS] Set kafka clientId while fetch messages Closesapache#18414 - [SPARK-21169] [core] Make sure to update application status to RUNNING if executors are accepted and RUNNING after recovery Closesapache#18432 - resolve com.esotericsoftware.kryo.KryoException Closesapache#18490 - [SPARK-21269][Core][WIP] Fix FetchFailedException when enable maxReqSizeShuffleToMem and KryoSerializer Closesapache#18585 - SPARK-21359 Closesapache#18609 - Spark SQL merge small files to big files Update InsertIntoHiveTable.scala Added: Closesapache#18308 - [SPARK-21099][Spark Core] INFO Log Message Using Incorrect Executor I… Closesapache#18599 - [SPARK-21372] spark writes one log file even I set the number of spark_rotate_log to 0 Closesapache#18619 - [SPARK-21397][BUILD]Maven shade plugin adding dependency-reduced-pom.xml to … Closesapache#18667 - Fix the simpleString used in error messages Closesapache#18782 - Branch 2.1 Added: Closesapache#17694 - [SPARK-12717][PYSPARK] Resolving race condition with pyspark broadcasts when using multiple threads Added: Closesapache#16456 - [SPARK-18994] clean up the local directories for application in future by annother thread Closesapache#18683 - [SPARK-21474][CORE] Make number of parallel fetches from a reducer configurable Closesapache#18690 - [SPARK-21334][CORE] Add metrics reporting service to External Shuffle Server Added: Closesapache#18827 - Merge pull request 1 from apache/master ## How was this patch tested? N/A Author: hyukjinkwon <gurwls223@gmail.com> Closesapache#18780 from HyukjinKwon/close-prs.
What changes were proposed in this pull request?
This PR is for adding an parameter (spark.input.dir.recursive) to control the value of recursive in SparkContext#addFile, so we can support "add file hdfs://dir/path" cmd in SparkSQL
How was this patch tested?
manual tests:
set the conf: --conf spark.input.dir.recursive=true, and run spark-sql -e "add file hdfs://dir/path"