[SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now by HyukjinKwon · Pull Request #24644 · apache/spark

HyukjinKwon · 2019-05-20T02:36:10Z

What changes were proposed in this pull request?

This PR excludes 'hive-thriftserver' in modules to test for hadoop3.2 for now as well

How was this patch tested?

Manually tested via run-tests.py

HyukjinKwon · 2019-05-20T02:37:10Z

 >>> x = [x.name for x in determine_modules_to_test([modules.sql])]
 >>> x # doctest: +NORMALIZE_WHITESPACE
+ ... # doctest: +SKIP
 ['sql', 'avro', 'hive', 'mllib', 'sql-kafka-0-10', 'examples', 'hive-thriftserver',


l manually tested with HADOOP_PROFILE=hadoop3.2 python -m doctest run-tests.py and python -m doctest run-tests.py

HyukjinKwon · 2019-05-20T02:37:37Z

+ hadoop_version = os.environ.get("HADOOP_PROFILE", "hadoop2.7")
+if hadoop_version == "hadoop3.2":
+ print("[info] Skip unsupported module:", "hive-thriftserver")
+ all_modules = [m for m in all_modules if m.name != "hive-thriftserver"]


Similarily, I tested all_modules. This is just to move the codes to remove later in single place.

This just does the same thing as before, right?

yea I just moved.

Got it. Thank you.

dongjoon-hyun · 2019-05-20T02:44:13Z

Thank you so much, @HyukjinKwon !

This reverts commit fd05eaf.

dongjoon-hyun · 2019-05-20T02:48:48Z

-else:
- hadoop_version = os.environ.get("HADOOP_PROFILE", "hadoop2.7")
-print("[info] Choosing supported modules with Hadoop profile", hadoop_version)
-


Since this is a partial revert of SPARK-27402, ping @wangyum .

wangyum · 2019-05-20T02:49:03Z

+changed_modules = toposort_flatten(
 {m: set(m.dependencies).intersection(modules_to_test) for m in modules_to_test}, sort=True)

+ # TODO: Skip hive-thriftserver module for hadoop-3.2. remove this once hadoop-3.2 support it


Could we revert modules.py to https://github.com/apache/spark/blob/3729efb4d0420700a396c79a83a1d5db25ac3bcb/dev/sparktestsupport/modules.py and just keep this change is enough?

Just keeping this change isn't enough because build and some other checks use profiles from root's build_profile_flags.

OK. Thank you @HyukjinKwon

To be clear, this fix itself is a temp fix too in order to make the affected scope isolated and minimised. To fix it properly, we should maybe be able to fix it within Module at modules.py (and remove this entire module conditionally)

HyukjinKwon · 2019-05-20T03:13:38Z

profiles look fine so far. I will switch to SBT's to make sure it doesn't affect our regular builds.

HyukjinKwon · 2019-05-20T03:22:49Z

retest this please

HyukjinKwon · 2019-05-20T03:25:27Z

retest this please

HyukjinKwon · 2019-05-20T03:53:12Z

okie profiles look good.

SparkQA · 2019-05-20T04:45:39Z

Test build #105544 has finished for PR 24644 at commit 92abff8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-05-20T05:06:13Z

Test build #105545 has finished for PR 24644 at commit 92abff8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-05-20T05:25:42Z

retest this please

SparkQA · 2019-05-20T06:43:42Z

Test build #105543 has finished for PR 24644 at commit 92abff8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-05-20T06:47:22Z

Test build #105542 has finished for PR 24644 at commit fd05eaf.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-05-20T06:49:22Z

@HyukjinKwon May be need add [test-maven] in title because of HIVE-21639.

SparkQA · 2019-05-20T06:49:53Z

Test build #105541 has finished for PR 24644 at commit 995ef1a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-05-20T06:53:57Z

Test build #105547 has finished for PR 24644 at commit 92abff8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-05-20T07:08:05Z

Do you mean regular PR build with hadoop 2.7 is affected? I am trying to see if this PR affects regular build with hadoop 2.7

HyukjinKwon · 2019-05-20T07:08:24Z

retest this please

wangyum · 2019-05-20T07:17:26Z

hadoop-3.2 needs [test-maven]. hadoop-2.7 does not [test-maven].

HyukjinKwon · 2019-05-20T07:18:08Z

yea thanks for clarification.

SparkQA · 2019-05-20T09:10:22Z

Test build #105557 has finished for PR 24644 at commit 92abff8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-05-20T09:45:47Z

Okie. I think it's ready if I am not mistaken.

dongjoon-hyun

+1, LGTM. Merged to master.

HyukjinKwon · 2019-05-20T15:34:17Z

Thanks, @dongjoon-hyun

Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now
995ef1a

HyukjinKwon commented May 20, 2019
View reviewed changes

HyukjinKwon changed the title ~~[SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now~~[SPARK-27402][INFRA][FOLLOW-UP][test-hadoop3.2][test-maven] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for nowMay 20, 2019

HyukjinKwon mentioned this pull request May 20, 2019
[SPARK-27699][FOLLOW-UP][SQL][test-hadoop3.2][test-maven] Fix hadoop-3.2 test error #24639
Closed

Change in SQL to check the module discovery
fd05eaf

Revert "Change in SQL to check the module discovery"
92abff8
This reverts commit fd05eaf.

dongjoon-hyun reviewed May 20, 2019
View reviewed changes

Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala Outdated

dongjoon-hyun reviewed May 20, 2019
View reviewed changes

wangyum reviewed May 20, 2019
View reviewed changes

HyukjinKwon changed the title ~~[SPARK-27402][INFRA][FOLLOW-UP][test-hadoop3.2][test-maven] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now~~[SPARK-27402][INFRA][FOLLOW-UP][test-hadoop3.2] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for nowMay 20, 2019

HyukjinKwon changed the title ~~[SPARK-27402][INFRA][FOLLOW-UP][test-hadoop3.2] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for now~~[SPARK-27402][INFRA][FOLLOW-UP] Exclude 'hive-thriftserver' in modules to test for hadoop3.2 for nowMay 20, 2019

dongjoon-hyun approved these changes May 20, 2019
View reviewed changes

dongjoon-hyun closed this in b7bf4fdMay 20, 2019

HyukjinKwon deleted the SPARK-27402 branch March 3, 2020 01:19

Uh oh!

Conversation

HyukjinKwon commented May 20, 2019

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented May 20, 2019

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwonMay 20, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented May 20, 2019

Uh oh!

HyukjinKwon commented May 20, 2019

Uh oh!

HyukjinKwon commented May 20, 2019

Uh oh!

HyukjinKwon commented May 20, 2019

Uh oh!

SparkQA commented May 20, 2019

Uh oh!

SparkQA commented May 20, 2019

Uh oh!

HyukjinKwon commented May 20, 2019

Uh oh!

SparkQA commented May 20, 2019

Uh oh!

SparkQA commented May 20, 2019

Uh oh!

wangyum commented May 20, 2019

Uh oh!

SparkQA commented May 20, 2019

Uh oh!

SparkQA commented May 20, 2019

Uh oh!

HyukjinKwon commented May 20, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented May 20, 2019

Uh oh!

wangyum commented May 20, 2019

Uh oh!

HyukjinKwon commented May 20, 2019

Uh oh!

SparkQA commented May 20, 2019

Uh oh!

HyukjinKwon commented May 20, 2019

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented May 20, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

HyukjinKwonMay 20, 2019•
edited
Loading

HyukjinKwon commented May 20, 2019•
edited
Loading