Skip to content

[SPARK-19490][SQL] ignore case sensitivity when filtering hive partition columns - #16832

Closed
cenyuhai wants to merge 9 commits into
apache:masterfrom
cenyuhai:SPARK-19490
Closed

[SPARK-19490][SQL] ignore case sensitivity when filtering hive partition columns#16832
cenyuhai wants to merge 9 commits into
apache:masterfrom
cenyuhai:SPARK-19490

Conversation

@cenyuhai

@cenyuhaicenyuhai commented Feb 7, 2017•
edited
Loading

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

ignore hive column names case-sensitive when comparing with partition columns

How was this patch tested?

@cenyuhaicenyuhai changed the title [SPARK-19490][SQL] change column names to lower case[SPARK-19490][SQL] change hive column names to lower caseFeb 7, 2017
@rxin

rxin commented Feb 7, 2017

Copy link
Copy Markdown
Contributor

hm is it safe to just do this change?

@SparkQA

SparkQA commented Feb 7, 2017

Copy link
Copy Markdown

Test build #72501 has finished for PR 16832 at commit efb0b4b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cenyuhai

cenyuhai commented Feb 10, 2017

Copy link
Copy Markdown
ContributorAuthor

@rxin I think it is safe, it is only used to check whether the schema contains the columns. Hive columns are not case-sensitive.

@SparkQA

SparkQA commented Feb 10, 2017

Copy link
Copy Markdown

Test build #72689 has started for PR 16832 at commit 6884ad3.

@cenyuhaicenyuhai changed the title [SPARK-19490][SQL] change hive column names to lower case[WIP][SPARK-19490][SQL] change hive column names to lower caseFeb 12, 2017
@cenyuhaicenyuhai changed the title [WIP][SPARK-19490][SQL] change hive column names to lower case[WIP][SPARK-19490][SQL] ignore hive column names case-sensitive when comparing with partition columnsFeb 12, 2017
@cenyuhaicenyuhai changed the title [WIP][SPARK-19490][SQL] ignore hive column names case-sensitive when comparing with partition columns[WIP][SPARK-19490][SQL] ignore case-sensitive when filtering hive partition columnsFeb 12, 2017
@cenyuhaicenyuhai changed the title [WIP][SPARK-19490][SQL] ignore case-sensitive when filtering hive partition columns[WIP][SPARK-19490][SQL] ignore case sensitivity when filtering hive partition columnsFeb 12, 2017
@SparkQA

SparkQA commented Feb 12, 2017

Copy link
Copy Markdown

Test build #72750 has finished for PR 16832 at commit 7214b8a.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Feb 12, 2017

Copy link
Copy Markdown

Test build #72755 has finished for PR 16832 at commit b734646.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Feb 12, 2017

Copy link
Copy Markdown

Test build #72758 has finished for PR 16832 at commit 358769c.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Feb 12, 2017

Copy link
Copy Markdown

Test build #72763 has started for PR 16832 at commit 5e91877.

@SparkQA

SparkQA commented Feb 22, 2017

Copy link
Copy Markdown

Test build #73267 has started for PR 16832 at commit f790821.

@SparkQA

SparkQA commented Feb 22, 2017

Copy link
Copy Markdown

Test build #73275 has finished for PR 16832 at commit c3f7444.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Feb 22, 2017

Copy link
Copy Markdown

Test build #73278 has finished for PR 16832 at commit c9dd09b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cenyuhaicenyuhai changed the title [WIP][SPARK-19490][SQL] ignore case sensitivity when filtering hive partition columns[SPARK-19490][SQL] ignore case sensitivity when filtering hive partition columnsFeb 23, 2017
@janua

janua commented Apr 26, 2017

Copy link
Copy Markdown

👍
Just bumped into this issue. The error message doesn't make it obvious what's wrong.

@taklwu

taklwu commented Jun 9, 2017

Copy link
Copy Markdown

any updates on this PR?

@cenyuhai

cenyuhai commented Jun 17, 2017

Copy link
Copy Markdown
ContributorAuthor

@taklwu this pr is completed, you can merge this pr by yourself. A committer told me that other pr has fixed this bug, my pr will not be merged......

@taklwu

taklwu commented Jun 17, 2017

Copy link
Copy Markdown

May I get the link of which commit/pr fix this issue? thanks a lot

@HyukjinKwonHyukjinKwon mentioned this pull request Jul 31, 2017
Closed
@asfgitasfgit closed this in 3a45c7fAug 5, 2017
@LantaoJin

LantaoJin commented Sep 15, 2017

Copy link
Copy Markdown
Contributor

Duplicates to SPARK-18572

zifeif2 pushed a commit to zifeif2/spark that referenced this pull request Nov 22, 2025
@HyukjinKwon@zifeif2
[INFRA] Close stale PRs
6639102
## What changes were proposed in this pull request?
This PR proposes to close stale PRs, mostly the same instances with apache#18017Closesapache#14085 - [SPARK-16408][SQL] SparkSQL Added file get Exception: is a directory …
Closesapache#14239 - [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism to accelerate shuffle stage.
Closesapache#14567 - [SPARK-16992][PYSPARK] Python Pep8 formatting and import reorganisation
Closesapache#14579 - [SPARK-16921][PYSPARK] RDD/DataFrame persist()/cache() should return Python context managers
Closesapache#14601 - [SPARK-13979][Core] Killed executor is re spawned without AWS key…
Closesapache#14830 - [SPARK-16992][PYSPARK][DOCS] import sort and autopep8 on Pyspark examples
Closesapache#14963 - [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in lint-python
Closesapache#15227 - [SPARK-17655][SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage
Closesapache#15240 - [SPARK-17556] [CORE] [SQL] Executor side broadcast for broadcast joins
Closesapache#15405 - [SPARK-15917][CORE] Added support for number of executors in Standalone [WIP]
Closesapache#16099 - [SPARK-18665][SQL] set statement state to "ERROR" after user cancel job
Closesapache#16445 - [SPARK-19043][SQL]Make SparkSQLSessionManager more configurable
Closesapache#16618 - [SPARK-14409][ML][WIP] Add RankingEvaluator
Closesapache#16766 - [SPARK-19426][SQL] Custom coalesce for Dataset
Closesapache#16832 - [SPARK-19490][SQL] ignore case sensitivity when filtering hive partition columns
Closesapache#17052 - [SPARK-19690][SS] Join a streaming DataFrame with a batch DataFrame which has an aggregation may not work
Closesapache#17267 - [SPARK-19926][PYSPARK] Make pyspark exception more user-friendly
Closesapache#17371 - [SPARK-19903][PYSPARK][SS] window operator miss the `watermark` metadata of time column
Closesapache#17401 - [SPARK-18364][YARN] Expose metrics for YarnShuffleService
Closesapache#17519 - [SPARK-15352][Doc] follow-up: add configuration docs for topology-aware block replication
Closesapache#17530 - [SPARK-5158] Access kerberized HDFS from Spark standalone
Closesapache#17854 - [SPARK-20564][Deploy] Reduce massive executor failures when executor count is large (>2000)
Closesapache#17979 - [SPARK-19320][MESOS][WIP]allow specifying a hard limit on number of gpus required in each spark executor when running on mesos
Closesapache#18127 - [SPARK-6628][SQL][Branch-2.1] Fix ClassCastException when executing sql statement 'insert into' on hbase table
Closesapache#18236 - [SPARK-21015] Check field name is not null and empty in GenericRowWit…
Closesapache#18269 - [SPARK-21056][SQL] Use at most one spark job to list files in InMemoryFileIndex
Closesapache#18328 - [SPARK-21121][SQL] Support changing storage level via the spark.sql.inMemoryColumnarStorage.level variable
Closesapache#18354 - [SPARK-18016][SQL][CATALYST][BRANCH-2.1] Code Generation: Constant Pool Limit - Class Splitting
Closesapache#18383 - [SPARK-21167][SS] Set kafka clientId while fetch messages
Closesapache#18414 - [SPARK-21169] [core] Make sure to update application status to RUNNING if executors are accepted and RUNNING after recovery
Closesapache#18432 - resolve com.esotericsoftware.kryo.KryoException
Closesapache#18490 - [SPARK-21269][Core][WIP] Fix FetchFailedException when enable maxReqSizeShuffleToMem and KryoSerializer
Closesapache#18585 - SPARK-21359
Closesapache#18609 - Spark SQL merge small files to big files Update InsertIntoHiveTable.scala
Added:
Closesapache#18308 - [SPARK-21099][Spark Core] INFO Log Message Using Incorrect Executor I…
Closesapache#18599 - [SPARK-21372] spark writes one log file even I set the number of spark_rotate_log to 0
Closesapache#18619 - [SPARK-21397][BUILD]Maven shade plugin adding dependency-reduced-pom.xml to …
Closesapache#18667 - Fix the simpleString used in error messages
Closesapache#18782 - Branch 2.1
Added:
Closesapache#17694 - [SPARK-12717][PYSPARK] Resolving race condition with pyspark broadcasts when using multiple threads
Added:
Closesapache#16456 - [SPARK-18994] clean up the local directories for application in future by annother thread
Closesapache#18683 - [SPARK-21474][CORE] Make number of parallel fetches from a reducer configurable
Closesapache#18690 - [SPARK-21334][CORE] Add metrics reporting service to External Shuffle Server
Added:
Closesapache#18827 - Merge pull request 1 from apache/master
## How was this patch tested?
N/A
Author: hyukjinkwon <gurwls223@gmail.com>
Closesapache#18780 from HyukjinKwon/close-prs.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@cenyuhai@rxin@SparkQA@janua@taklwu@LantaoJin