Skip to content

[SPARK-21397][BUILD]Maven shade plugin adding dependency-reduced-pom.xml to … - #18619

Closed
zuotingbing wants to merge 1 commit into
apache:masterfrom
zuotingbing:SPARK-21397
Closed

[SPARK-21397][BUILD]Maven shade plugin adding dependency-reduced-pom.xml to …#18619
zuotingbing wants to merge 1 commit into
apache:masterfrom
zuotingbing:SPARK-21397

Conversation

@zuotingbing

@zuotingbingzuotingbing commented Jul 13, 2017

Copy link
Copy Markdown

…base directory

What changes were proposed in this pull request?

  1. run ./build/mvn -DskipTests clean package and we get many files 'dependency-reduced-pom.xml'
    under the source directory

  2. run ./build/mvn clean, the files dependency-reduced-pom.xml can not be cleaned.

As a trivial Improvement, i suggest to specify dependencyReducedPomLocation to 'target/' to store these files.

How was this patch tested?

manual tests

@AmplabJenkins

Copy link
Copy Markdown

Can one of the admins verify this patch?

@srowen

Copy link
Copy Markdown
Member

I like the idea, but see this note: https://maven.apache.org/plugins/maven-shade-plugin/shade-mojo.html#dependencyReducedPomLocation . I'm not clear whether this has side-effects.

@zuotingbing

zuotingbing commented Jul 13, 2017

Copy link
Copy Markdown
Author

@srowen Thanks for your time. I am not sure the open issue, which is mentioned in the note, whether has bean resolved. as the note said

setting a value for this parameter with a directory other than ${basedir} will change the value of ${basedir}

,but after running mvn checkstyle:check , i get the right path of ${basedir}/target/checkstyle-output.xml

@srowen

Copy link
Copy Markdown
Member

That's not the problem this would create -- it changes basedir after execution of the shade plugin, so it might affect things like the release build. Is it really a problem? gitignore already ignores these files, even though they're a little annoying.

@zuotingbing

Copy link
Copy Markdown
Author

I just use mvn checkstyle:check ,which contains output files with basedir, to check whether the basedir is correct with this PR.
Could you please specify the affect exactly as you said? what is your mean about release build? or does it the output package is not right?

@zuotingbing

zuotingbing commented Jul 14, 2017

Copy link
Copy Markdown
Author

Yes this PR is a trivial Improvement for me. Thanks.

@srowen

Copy link
Copy Markdown
Member

Meaning I think you'd have to run the release process to verify the artifacts still work, because it's the process that uses shading

@srowensrowen mentioned this pull request Jul 31, 2017
zifeif2 pushed a commit to zifeif2/spark that referenced this pull request Nov 22, 2025
## What changes were proposed in this pull request?
This PR proposes to close stale PRs, mostly the same instances with apache#18017Closesapache#14085 - [SPARK-16408][SQL] SparkSQL Added file get Exception: is a directory …
Closesapache#14239 - [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism to accelerate shuffle stage.
Closesapache#14567 - [SPARK-16992][PYSPARK] Python Pep8 formatting and import reorganisation
Closesapache#14579 - [SPARK-16921][PYSPARK] RDD/DataFrame persist()/cache() should return Python context managers
Closesapache#14601 - [SPARK-13979][Core] Killed executor is re spawned without AWS key…
Closesapache#14830 - [SPARK-16992][PYSPARK][DOCS] import sort and autopep8 on Pyspark examples
Closesapache#14963 - [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in lint-python
Closesapache#15227 - [SPARK-17655][SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage
Closesapache#15240 - [SPARK-17556] [CORE] [SQL] Executor side broadcast for broadcast joins
Closesapache#15405 - [SPARK-15917][CORE] Added support for number of executors in Standalone [WIP]
Closesapache#16099 - [SPARK-18665][SQL] set statement state to "ERROR" after user cancel job
Closesapache#16445 - [SPARK-19043][SQL]Make SparkSQLSessionManager more configurable
Closesapache#16618 - [SPARK-14409][ML][WIP] Add RankingEvaluator
Closesapache#16766 - [SPARK-19426][SQL] Custom coalesce for Dataset
Closesapache#16832 - [SPARK-19490][SQL] ignore case sensitivity when filtering hive partition columns
Closesapache#17052 - [SPARK-19690][SS] Join a streaming DataFrame with a batch DataFrame which has an aggregation may not work
Closesapache#17267 - [SPARK-19926][PYSPARK] Make pyspark exception more user-friendly
Closesapache#17371 - [SPARK-19903][PYSPARK][SS] window operator miss the `watermark` metadata of time column
Closesapache#17401 - [SPARK-18364][YARN] Expose metrics for YarnShuffleService
Closesapache#17519 - [SPARK-15352][Doc] follow-up: add configuration docs for topology-aware block replication
Closesapache#17530 - [SPARK-5158] Access kerberized HDFS from Spark standalone
Closesapache#17854 - [SPARK-20564][Deploy] Reduce massive executor failures when executor count is large (>2000)
Closesapache#17979 - [SPARK-19320][MESOS][WIP]allow specifying a hard limit on number of gpus required in each spark executor when running on mesos
Closesapache#18127 - [SPARK-6628][SQL][Branch-2.1] Fix ClassCastException when executing sql statement 'insert into' on hbase table
Closesapache#18236 - [SPARK-21015] Check field name is not null and empty in GenericRowWit…
Closesapache#18269 - [SPARK-21056][SQL] Use at most one spark job to list files in InMemoryFileIndex
Closesapache#18328 - [SPARK-21121][SQL] Support changing storage level via the spark.sql.inMemoryColumnarStorage.level variable
Closesapache#18354 - [SPARK-18016][SQL][CATALYST][BRANCH-2.1] Code Generation: Constant Pool Limit - Class Splitting
Closesapache#18383 - [SPARK-21167][SS] Set kafka clientId while fetch messages
Closesapache#18414 - [SPARK-21169] [core] Make sure to update application status to RUNNING if executors are accepted and RUNNING after recovery
Closesapache#18432 - resolve com.esotericsoftware.kryo.KryoException
Closesapache#18490 - [SPARK-21269][Core][WIP] Fix FetchFailedException when enable maxReqSizeShuffleToMem and KryoSerializer
Closesapache#18585 - SPARK-21359
Closesapache#18609 - Spark SQL merge small files to big files Update InsertIntoHiveTable.scala
Added:
Closesapache#18308 - [SPARK-21099][Spark Core] INFO Log Message Using Incorrect Executor I…
Closesapache#18599 - [SPARK-21372] spark writes one log file even I set the number of spark_rotate_log to 0
Closesapache#18619 - [SPARK-21397][BUILD]Maven shade plugin adding dependency-reduced-pom.xml to …
Closesapache#18667 - Fix the simpleString used in error messages
Closesapache#18782 - Branch 2.1
Added:
Closesapache#17694 - [SPARK-12717][PYSPARK] Resolving race condition with pyspark broadcasts when using multiple threads
Added:
Closesapache#16456 - [SPARK-18994] clean up the local directories for application in future by annother thread
Closesapache#18683 - [SPARK-21474][CORE] Make number of parallel fetches from a reducer configurable
Closesapache#18690 - [SPARK-21334][CORE] Add metrics reporting service to External Shuffle Server
Added:
Closesapache#18827 - Merge pull request 1 from apache/master
## How was this patch tested?
N/A
Author: hyukjinkwon <gurwls223@gmail.com>
Closesapache#18780 from HyukjinKwon/close-prs.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@zuotingbing@AmplabJenkins@srowen