[SPARK-1636][MLLIB] Move main methods to examples by mengxr · Pull Request #584 · apache/spark

mengxr · 2014-04-28T23:29:00Z

NaiveBayes -> SparseNaiveBayes
KMeans -> DenseKMeans
SVMWithSGD and LogisticRegerssionWithSGD -> BinaryClassification
ALS -> MovieLensALS
LinearRegressionWithSGD, LassoWithSGD, and RidgeRegressionWithSGD -> LinearRegression
DecisionTree -> DecisionTreeRunner

scopt is used for parsing command-line parameters. scopt has MIT license and it only depends on scala-library.

Example help message:

BinaryClassification: an example app for binary classification.
Usage: BinaryClassification [options] <input>
--numIterations <value>
number of iterations
--stepSize <value>
initial step size, default: 1.0
--algorithm <value>
algorithm (SVM,LR), default: LR
--regType <value>
regularization type (L1,L2), default: L2
--regParam <value>
regularization parameter, default: 0.1
<input>
input paths to labeled examples in LIBSVM format

…egression example

AmplabJenkins · 2014-04-28T23:32:57Z

Merged build triggered.

AmplabJenkins · 2014-04-28T23:33:05Z

Merged build started.

AmplabJenkins · 2014-04-29T00:24:00Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-04-29T00:24:00Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14545/

rxin · 2014-04-29T03:23:32Z

I think ti is better to declare a main rather than extending App. It is more familiar to readers from the Java/C land.

rxin · 2014-04-29T03:34:34Z

LGTM. If you can avoid wild card imports for some packages, that'd be great.

AmplabJenkins · 2014-04-29T05:07:57Z

Merged build triggered.

AmplabJenkins · 2014-04-29T05:08:06Z

Merged build started.

mengxr · 2014-04-29T05:25:50Z

Made imports explicit except SparkContext and enums.

AmplabJenkins · 2014-04-29T05:27:57Z

Merged build triggered.

AmplabJenkins · 2014-04-29T05:28:06Z

Merged build started.

AmplabJenkins · 2014-04-29T05:47:57Z

Merged build triggered.

AmplabJenkins · 2014-04-29T05:48:07Z

Merged build started.

AmplabJenkins · 2014-04-29T06:38:57Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-04-29T06:38:57Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-04-29T06:38:57Z

Merged build finished.

AmplabJenkins · 2014-04-29T06:38:58Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14554/

AmplabJenkins · 2014-04-29T06:38:58Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14555/

AmplabJenkins · 2014-04-29T06:38:58Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14553/

rxin · 2014-04-29T07:41:07Z

Thanks. Merged!

* `NaiveBayes` -> `SparseNaiveBayes` * `KMeans` -> `DenseKMeans` * `SVMWithSGD` and `LogisticRegerssionWithSGD` -> `BinaryClassification` * `ALS` -> `MovieLensALS` * `LinearRegressionWithSGD`, `LassoWithSGD`, and `RidgeRegressionWithSGD` -> `LinearRegression` * `DecisionTree` -> `DecisionTreeRunner` `scopt` is used for parsing command-line parameters. `scopt` has MIT license and it only depends on `scala-library`. Example help message: ~~~ BinaryClassification: an example app for binary classification. Usage: BinaryClassification [options] <input> --numIterations <value> number of iterations --stepSize <value> initial step size, default: 1.0 --algorithm <value> algorithm (SVM,LR), default: LR --regType <value> regularization type (L1,L2), default: L2 --regParam <value> regularization parameter, default: 0.1 <input> input paths to labeled examples in LIBSVM format ~~~ Author: Xiangrui Meng <meng@databricks.com> Closes#584 from mengxr/mllib-main and squashes the following commits: 7b58c60 [Xiangrui Meng] minor 6e35d7e [Xiangrui Meng] make imports explicit and fix code style c6178c9 [Xiangrui Meng] update TS PCA/SVD to use new spark-submit 6acff75 [Xiangrui Meng] use scopt for DecisionTreeRunner be86069 [Xiangrui Meng] use main instead of extending App b3edf68 [Xiangrui Meng] move DecisionTree's main method to examples 8bfaa5a [Xiangrui Meng] change NaiveBayesParams to Params fe23dcb [Xiangrui Meng] remove main from KMeans and add DenseKMeans as an example 67f4448 [Xiangrui Meng] remove main methods from linear regression algorithms and add LinearRegression example b066bbc [Xiangrui Meng] remove main from ALS and add MovieLensALS example b040f3b [Xiangrui Meng] change BinaryClassificationParams to Params 577945b [Xiangrui Meng] remove unused imports from NB 3d299bc [Xiangrui Meng] remove main from LR/SVM and add an example app for binary classification f70878e [Xiangrui Meng] remove main from NaiveBayes and add an example NaiveBayes app 01ec2cd [Xiangrui Meng] Merge branch 'master' into mllib-main 9420692 [Xiangrui Meng] add scopt to examples dependencies (cherry picked from commit 3f38334) Signed-off-by: Reynold Xin <rxin@apache.org>

* `NaiveBayes` -> `SparseNaiveBayes` * `KMeans` -> `DenseKMeans` * `SVMWithSGD` and `LogisticRegerssionWithSGD` -> `BinaryClassification` * `ALS` -> `MovieLensALS` * `LinearRegressionWithSGD`, `LassoWithSGD`, and `RidgeRegressionWithSGD` -> `LinearRegression` * `DecisionTree` -> `DecisionTreeRunner` `scopt` is used for parsing command-line parameters. `scopt` has MIT license and it only depends on `scala-library`. Example help message: ~~~ BinaryClassification: an example app for binary classification. Usage: BinaryClassification [options] <input> --numIterations <value> number of iterations --stepSize <value> initial step size, default: 1.0 --algorithm <value> algorithm (SVM,LR), default: LR --regType <value> regularization type (L1,L2), default: L2 --regParam <value> regularization parameter, default: 0.1 <input> input paths to labeled examples in LIBSVM format ~~~ Author: Xiangrui Meng <meng@databricks.com> Closesapache#584 from mengxr/mllib-main and squashes the following commits: 7b58c60 [Xiangrui Meng] minor 6e35d7e [Xiangrui Meng] make imports explicit and fix code style c6178c9 [Xiangrui Meng] update TS PCA/SVD to use new spark-submit 6acff75 [Xiangrui Meng] use scopt for DecisionTreeRunner be86069 [Xiangrui Meng] use main instead of extending App b3edf68 [Xiangrui Meng] move DecisionTree's main method to examples 8bfaa5a [Xiangrui Meng] change NaiveBayesParams to Params fe23dcb [Xiangrui Meng] remove main from KMeans and add DenseKMeans as an example 67f4448 [Xiangrui Meng] remove main methods from linear regression algorithms and add LinearRegression example b066bbc [Xiangrui Meng] remove main from ALS and add MovieLensALS example b040f3b [Xiangrui Meng] change BinaryClassificationParams to Params 577945b [Xiangrui Meng] remove unused imports from NB 3d299bc [Xiangrui Meng] remove main from LR/SVM and add an example app for binary classification f70878e [Xiangrui Meng] remove main from NaiveBayes and add an example NaiveBayes app 01ec2cd [Xiangrui Meng] Merge branch 'master' into mllib-main 9420692 [Xiangrui Meng] add scopt to examples dependencies

apache#522 got messed after i rewrote the branch hadoop_jar_name. So created a new one. Author: Bijay Bisht <bijay.bisht@gmail.com> Closesapache#584 from bijaybisht/hadoop_jar_name_on_0.9.0 and squashes the following commits: 1b6fb3c [Bijay Bisht] Ported hadoopClient jar for < 1.0.1 fix

Clean up the pre-built k8s binaries output folder which is generated in `install-k8s` role. Otherwise `kubetest` will fail due to some unknown conflicts.

mengxr added 11 commits April 28, 2014 10:21

add scopt to examples dependencies
9420692

Merge branch 'master' into mllib-main
01ec2cd

remove main from NaiveBayes and add an example NaiveBayes app
f70878e

remove main from LR/SVM and add an example app for binary classification
3d299bc

remove unused imports from NB
577945b

change BinaryClassificationParams to Params
b040f3b

remove main from ALS and add MovieLensALS example
b066bbc

remove main methods from linear regression algorithms and add LinearR…
67f4448
…egression example

remove main from KMeans and add DenseKMeans as an example
fe23dcb

change NaiveBayesParams to Params
8bfaa5a

move DecisionTree's main method to examples
b3edf68

mengxr mentioned this pull request Apr 28, 2014
SPARK-1637: Clean up examples for 1.0 #571
Closed
3 tasks

rxin reviewed Apr 29, 2014
View reviewed changes

use main instead of extending App
be86069

mengxr added 2 commits April 28, 2014 21:44

use scopt for DecisionTreeRunner
6acff75

update TS PCA/SVD to use new spark-submit
c6178c9

make imports explicit and fix code style
6e35d7e

minor
7b58c60

asfgit closed this in 3f38334Apr 29, 2014

mengxr deleted the mllib-main branch May 7, 2014 00:06

peter-toth mentioned this pull request Mar 25, 2020
[SPARK-29375][SQL] Exchange reuse across all subquery levels #26044
Closed

arjunshroff pushed a commit to arjunshroff/spark that referenced this pull request Nov 24, 2020
MapR [SPARK-517] Can't use ssl via spark beeline (apache#584)
3d00f35

Uh oh!

Conversation

mengxr commented Apr 28, 2014

Uh oh!

AmplabJenkins commented Apr 28, 2014

Uh oh!

AmplabJenkins commented Apr 28, 2014

Uh oh!

AmplabJenkins commented Apr 29, 2014

Uh oh!

AmplabJenkins commented Apr 29, 2014

Uh oh!

rxinApr 29, 2014

Choose a reason for hiding this comment

Uh oh!

mengxrApr 29, 2014

Choose a reason for hiding this comment

Uh oh!

rxin commented Apr 29, 2014

Uh oh!

AmplabJenkins commented Apr 29, 2014

Uh oh!

AmplabJenkins commented Apr 29, 2014

Uh oh!

mengxr commented Apr 29, 2014

Uh oh!

AmplabJenkins commented Apr 29, 2014

Uh oh!

AmplabJenkins commented Apr 29, 2014

Uh oh!

AmplabJenkins commented Apr 29, 2014

Uh oh!

AmplabJenkins commented Apr 29, 2014

Uh oh!

AmplabJenkins commented Apr 29, 2014

Uh oh!

AmplabJenkins commented Apr 29, 2014

Uh oh!

AmplabJenkins commented Apr 29, 2014

Uh oh!

AmplabJenkins commented Apr 29, 2014

Uh oh!

AmplabJenkins commented Apr 29, 2014

Uh oh!

AmplabJenkins commented Apr 29, 2014

Uh oh!

rxin commented Apr 29, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants