[SPARK-8277][SPARKR] Faster createDataFrame using mapply

[saurfang]

With a single loop using mapply, I'm able to create DataFrame much faster from R data.frame.

Please see benchmark results and code:

Unit:millisecondsexprminlqmeanmedianuqmaxnevalold284.874542312.012427426.822889336.360288436.473771356.91518100new4.8750896.3576659.0134426.90472910.4159742.41479100

library(nycflights13)
library(microbenchmark)
data<- head(flights, n=1000)
# get rid of factor typedropFactor<-function(x) {
if (is.factor(x)) {
as.character(x)
} else {
x
}
}
createDataFrameNew<-function(data) {
do.call(mapply, c(list, unname(lapply(data, dropFactor)), SIMPLIFY=FALSE))
}
createDataFrameOld<-function(data) {
n<- nrow(data)
m<- ncol(data)
lapply(1:n, function(i) {
lapply(1:m, function(j) { dropFactor(data[i,j]) })
})
}
my_check<-function(values) {
all(sapply(values[-1], function(x) identical(values[[1]], x)))
}
microbenchmark(old= createDataFrameOld(data), new= createDataFrameNew(data), check=my_check)

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44181/
Test FAILed.

[felixcheung]

Hi thanks for the contribution, you might want to check out the ongoing work in #9099
and SPARK-11086

[saurfang]

Ah. Thanks for the pointer and I didn't realize this issue has already been worked on. Looks like that PR already had all my brilliant idea ;) I'm closing this then. My apology on the duplicate work.