[SPARK-20727] Skip tests that use Hadoop utils on CRAN Windows

[shivaram]

What changes were proposed in this pull request?

This change skips tests that use the Hadoop libraries while running
on CRAN check with Windows as the operating system. This is to handle
cases where the Hadoop winutils binaries are missing on the target
system. The skipped tests consist of

1. Tests that save, load a model in MLlib
2. Tests that save, load CSV, JSON and Parquet files in SQL
3. Hive tests

How was this patch tested?

Tested by running on a local windows VM with HADOOP_HOME unset. Also testing with https://win-builder.r-project.org

[shivaram]

cc @felixcheung

FWIW it might easier to view the diff by adding w=1 to the URL. i.e. https://github.com/apache/spark/pull/17966/files?w=1

[vanzin]

This doesn't seem to really have anything to do with the errors in SPARK-20666. As I pointed out, they do not only happen on Windows.

[shivaram]

Sorry @vanzin I got the wrong JIRA number. Fixing it now

[shivaram]

This is SPARK-20727 - I just happened to have the other JIRA also open and pasted it incorrectly

[SparkQA]

Test build #76876 has finished for PR 17966 at commit 40273ab.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[shivaram]

Actually thinking more about this, I think we should be checking for availability of hadoop library / binaries rather than is_cran. For example I just found that win-builder only runs R CMD check (no --as-cran) and so this PR doesn't actually help on that machine.

[shivaram]

@HyukjinKwon Do we know why things sometime queue for a long time on AppVeyor ? Like this PR has been queued for around 5 hours right now.

[HyukjinKwon]

Thank you for cc'ing me. I think primarily it is because single AppVeyor account is shared across several Apache projects but the number of concurrent jobs is single up to my knowledge. So, it is a kind of a global queue.

I am monitoring ...

https://ci.appveyor.com/project/ApacheSoftwareFoundation/spark
https://ci.appveyor.com/project/ApacheSoftwareFoundation/reef
https://ci.appveyor.com/project/ApacheSoftwareFoundation/nifi
https://ci.appveyor.com/project/ApacheSoftwareFoundation/thrift
https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow
https://ci.appveyor.com/project/ApacheSoftwareFoundation/syncope

[HyukjinKwon]

Just FYI, closing and opening a PR is a workaround to re-trigger the build in AppVeyor as (I assume) we all don't currently have the permission via AppVeyor Web UI.

[felixcheung]

Probably - but how to check for Hadoop? See if HADOOP_HOME is set? We don't need to set that on *nix though, I think

[felixcheung]

So I'd propose this

is_cran <- function() { !identical(Sys.getenv("NOT_CRAN"), "true") }
is_windows <- function() { .Platform$OS.type == "windows" }
hadoop_home_set <- function() { !identical(Sys.getenv("HADOOP_HOME"), "") }
not_cran_and_windows_with_hadoop <- function() { !is_cran() && (!is_windows() || hadoop_home_set()) }

[felixcheung]

@shivaram have you got a chance to work on this again?

[shivaram]

Sorry I've been out traveling -- I'll try to update this by tonight

[shivaram]

@felixcheung I made the change - I'm right now going to test this in my Windows VM. Will update this PR with the results

[SparkQA]

Test build #77114 has finished for PR 17966 at commit 7555501.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[shivaram]

@felixcheung Unfortunately I'm out traveling and haven't been able to do the windows tests yet -- Would you have a chance to do that ? Also what are your thoughts on merging this while we test given the upcoming RC cut ?

[felixcheung]

testing now - could you submit to https://win-builder.r-project.org?

[felixcheung]

LGTM
tested package with and without HADOOP_HOME, on Windows, both passed

[felixcheung]

merged to master/2.2

I think we should still check win-builder. Also it's a bit hard to tell if the skipped tests are skipped - might want to follow up with a trace

[shivaram]

Thanks I'll try to kick off the winbuilder build soon (i'm out of town till tomorrow). One more thing we might need to fix is that winbuilder has a 10 or 20 minute time limit for tests (not sure if the CRAN check has a similar limit ?)