Skip to content

[SPARK-14585][ML][WIP] Provide accessor methods for Pipeline stages - #12420

Closed
pravingadakh wants to merge 1 commit into
apache:masterfrom
pravingadakh:SPARK-14585
Closed

[SPARK-14585][ML][WIP] Provide accessor methods for Pipeline stages#12420
pravingadakh wants to merge 1 commit into
apache:masterfrom
pravingadakh:SPARK-14585

Conversation

@pravingadakh

@pravingadakhpravingadakh commented Apr 15, 2016

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This PR adds accessor methods for pipeline stages in Pipeline and PipelineModel classes.

@jkbradley Hi, please validate if this is something that you had in mind. If so, I can start on test cases for these.

@jkbradley

jkbradley commented Apr 20, 2016

Copy link
Copy Markdown
Member

test this please

@SparkQA

SparkQA commented Apr 20, 2016

Copy link
Copy Markdown

Test build #56401 has finished for PR 12420 at commit a4592c6.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jkbradley

jkbradley commented Apr 20, 2016

Copy link
Copy Markdown
Member

@pravingadakh Thanks! This is what I had in mind. In your tests, could you please add Java unit tests too to make sure these are Java-friendly?

Also, please check the type and throw exceptions when the type does not match. I'd recommend throwing IllegalArgumentException and specifying the expected and actual types.

jkbradley
jkbradley reviewed Apr 20, 2016
View reviewed changes
Comment threadmllib/src/main/scala/org/apache/spark/ml/Pipeline.scala

/** Returns stage at index i in Pipeline */
@Since("2.0.0")
def getStage[T <: PipelineStage](i: Int): T = getStages.apply(i).asInstanceOf[T]

@jkbradleyjkbradleyApr 20, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here and elsewhere, I would just write array(i) instead of array.apply(i)

Also, can you please document the @tparam?

@pravingadakh

pravingadakh commented Apr 23, 2016

Copy link
Copy Markdown
ContributorAuthor

@jkbradley I'll update the PR accordingly.

@jkbradley

jkbradley commented Apr 26, 2016

Copy link
Copy Markdown
Member

By the way, we'll need to get this merged quickly for it to make 2.0. (Thanks!)

@pravingadakh

pravingadakh commented Apr 28, 2016

Copy link
Copy Markdown
ContributorAuthor

@jkbradley I'll update it soon.

@pravingadakh

pravingadakh commented May 2, 2016

Copy link
Copy Markdown
ContributorAuthor

@jkbradley I have stumbled across an issue, would really appreciate your input. Have a look at this method:

def getStage[T <: PipelineStage : ClassTag](i: Int): T = {
getStages(i) match {
case stage: T => stage
case stage => throw new IllegalArgumentException(
s"Expected stage of type ${stage.getClass.getName}"
)
}
}

How do I avoid usage of ClassTag here (since those are not so java friendly)? Also I'm a bit skeptical about the error message.

@AmplabJenkins

AmplabJenkins commented Nov 27, 2016

Copy link
Copy Markdown

Can one of the admins verify this patch?

srowen
srowen requested changes Nov 27, 2016
View reviewed changes
Comment threadmllib/src/main/scala/org/apache/spark/ml/Pipeline.scala
def getStages: Array[PipelineStage] = $(stages).clone()

/** Returns stage at index i in Pipeline */
@Since("2.0.0")

@srowensrowenNov 27, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't be since 2.0.0 at this point. Also use a @return tag in the docs.

Comment threadmllib/src/main/scala/org/apache/spark/ml/Pipeline.scala
/** Returns all stages of this type */
@Since("2.0.0")
def getStagesOfType[T <: PipelineStage]: Array[T] = {
getStages.collect {

@srowensrowenNov 27, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: This is probably more natural as a one-liner

@jkbradley

jkbradley commented Dec 22, 2016

Copy link
Copy Markdown
Member

@pravingadakh Will you be able to continue with this? Thanks!

@jkbradley

jkbradley commented Feb 2, 2017

Copy link
Copy Markdown
Member

I missed the ClassTag question above. Let me take a look

@jkbradley

jkbradley commented Feb 2, 2017

Copy link
Copy Markdown
Member

Well, after spending a while looking around, I haven't found a good way to write this and make it Java friendly (i.e., not use ClassTag, Type, or TypeTag). Does anyone else have ideas? I'll try to return to this before long...

@HyukjinKwon

HyukjinKwon commented May 11, 2017

Copy link
Copy Markdown
Member

Hi @@pravingadakh, is this still active?

@HyukjinKwonHyukjinKwon mentioned this pull request May 17, 2017
Closed
@pravingadakh
pravingadakh deleted the SPARK-14585 branch July 24, 2017 18:50
zifeif2 pushed a commit to zifeif2/spark that referenced this pull request Nov 22, 2025
@HyukjinKwon@zifeif2
[INFRA] Close stale PRs
47a2872
## What changes were proposed in this pull request?
This PR proposes to close PRs ...
- inactive to the review comments more than a month
- WIP and inactive more than a month
- with Jenkins build failure but inactive more than a month
- suggested to be closed and no comment against that
- obviously looking inappropriate (e.g., Branch 0.5)
To make sure, I left a comment for each PR about a week ago and I could not have a response back from the author in these PRs below:
Closesapache#11129Closesapache#12085Closesapache#12162Closesapache#12419Closesapache#12420Closesapache#12491Closesapache#13762Closesapache#13837Closesapache#13851Closesapache#13881Closesapache#13891Closesapache#13959Closesapache#14091Closesapache#14481Closesapache#14547Closesapache#14557Closesapache#14686Closesapache#15594Closesapache#15652Closesapache#15850Closesapache#15914Closesapache#15918Closesapache#16285Closesapache#16389Closesapache#16652Closesapache#16743Closesapache#16893Closesapache#16975Closesapache#17001Closesapache#17088Closesapache#17119Closesapache#17272Closesapache#17971
Added:
Closesapache#17778Closesapache#17303Closesapache#17872
## How was this patch tested?
N/A
Author: hyukjinkwon <gurwls223@gmail.com>
Closesapache#18017 from HyukjinKwon/close-inactive-prs.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@srowensrowensrowen requested changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@pravingadakh@jkbradley@SparkQA@AmplabJenkins@HyukjinKwon@srowen