[MINOR][DOC] Update deprecation notes on Python/Hadoop/Scala.

[dongjoon-hyun]

What changes were proposed in this pull request?

We had better update the deprecation notes about Python 2.6, Hadoop (before 2.6.5) and Scala 2.10 in 2.2.0-RC4 documentation. Since this is a doc only update, I think we can update the doc during publishing.

BEFORE (2.2.0-RC4)

AFTER

How was this patch tested?

Manual.

SKIP_API=1 jekyll build

[dongjoon-hyun]

Hi, @marmbrus .
Could you review this PR about 2.2.0?

[srowen]

I wasn't clear whether we really truly don't support 2.6 in this 2.2.0 release, given https://issues.apache.org/jira/browse/SPARK-12661 ?

[dongjoon-hyun]

Oh, I see. Then, may I ask this at mailing list to be more clearer?

[SparkQA]

Test build #77754 has finished for PR 18207 at commit 69774c1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[marmbrus]

/cc @JoshRosen

[JoshRosen]

As of #17355 we no longer test against Python 2.6. That doesn't mean that 2.6 won't work today, but there's nothing stopping 2.6 support from breaking in a future 2.2.x release because we are no longer testing against that release.

#17355 replaced our Python 2.6 testing environment with a Python 2.7 release, so we can now begin to use language features and libraries which are only available from 2.7 onwards (such as set and dictionary comprehensions).

Therefore, this documentation change looks correct to me.

[dongjoon-hyun]

Thank you for confirming, @JoshRosen .

[rxin]

I believe we still support Python 2.6, given Jenkins runs 2.6... There seems to be no point in removing that support this late in the release cycle.

[dongjoon-hyun]

Jenkins runs with ['python2.7', 'python3.4', 'pypy'] only, doesn't it? Also, this is a major release cycle with the other big changes. For me, removing Python 2.6 seems not to be proper with subsequent minor version release cycles like 2.2.1 and 2.2.2.

[JoshRosen]

@rxin, as of #17355 Jenkins is using Python 2.7 sourced from a virtualenv instead of Python 2.6. That patch was merged into master before branch-2.2 was cut.

[rxin]

OK great then we have officially deprecated it, haven't we?

[dongjoon-hyun]

Yep. Right. Then, could you officially resolve SPARK-12661, too?

[srowen]

It sounds like 2.6 is still more supported than not-support in 2.2.x, but we should call it unsupported for 2.3.x.

Let's say that Python 2.6 support was really removed for 2.3.0, rather than 2.2.0.
It's fine to add a note about Hadoop before 2.6 being unsupported in 2.2.0, that's true.

Then we can merge in master. It's not essential to merge this for 2.2 as the existing text appears to be not-wrong.

[dongjoon-hyun]

Did you test on Python 2.6 at RC4 testing? And it happens to be supported at 2.2.0?
Actually, Jenkins does not run the tests over two months (since 29th Mar). As @JoshRosen said, there is no guard anymore in order to guarantee still more supported on 2.2.1 or later.

[JoshRosen]

Even if Python 2.6 "works by accident" on 2.2.0 that should not impact this wording: as long as we marked Python 2.6 as deprecated in an earlier release then it should be fine to claim non support in 2.2.0.

…

On Tue, Jun 6, 2017, 8:25 AM Dongjoon Hyun ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In docs/index.md <#18207 (comment)>: > uses Scala {{site.SCALA_BINARY_VERSION}}. You will need to use a compatible Scala version ({{site.SCALA_BINARY_VERSION}}.x). -Note that support for Java 7 was removed as of Spark 2.2.0. +Note that support for Java 7, Python 2.6 and old Hadoop versions before 2.6.5 were removed as of Spark 2.2.0. Did you test on Python 2.6 at RC4 testing? And it happens to be supported at 2.2.0? Actually, Jenkins does not run the tests over two months (since 29th Mar). As @JoshRosen <https://github.com/joshrosen> said, there is no guard anymore in order to guarantee still more supported on 2.2.1 or later. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#18207 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADGPJtuOTgL3_uYB4wzbbBpEBmlb1o2ks5sBW99gaJpZM4NweCt> .

[JoshRosen]

Put another way: if we found a bug only impacting Python 2.6 in 2.2.0 would we fix that bug in 2.2.1? If not then the right thing to do is to say 2.6 isn't supported.

…

On Tue, Jun 6, 2017, 8:44 AM Josh Rosen ***@***.***> wrote: Even if Python 2.6 "works by accident" on 2.2.0 that should not impact this wording: as long as we marked Python 2.6 as deprecated in an earlier release then it should be fine to claim non support in 2.2.0. On Tue, Jun 6, 2017, 8:25 AM Dongjoon Hyun ***@***.***> wrote: > ***@***.**** commented on this pull request. > ------------------------------ > > In docs/index.md > <#18207 (comment)>: > > > uses Scala {{site.SCALA_BINARY_VERSION}}. You will need to use a compatible Scala version > ({{site.SCALA_BINARY_VERSION}}.x). > > -Note that support for Java 7 was removed as of Spark 2.2.0. > +Note that support for Java 7, Python 2.6 and old Hadoop versions before 2.6.5 were removed as of Spark 2.2.0. > > Did you test on Python 2.6 at RC4 testing? And it happens to be supported > at 2.2.0? > Actually, Jenkins does not run the tests over two months (since 29th > Mar). As @JoshRosen <https://github.com/joshrosen> said, there is no > guard anymore in order to guarantee still more supported on 2.2.1 or > later. > > — > You are receiving this because you were mentioned. > > > Reply to this email directly, view it on GitHub > <#18207 (comment)>, or mute > the thread > <https://github.com/notifications/unsubscribe-auth/AADGPJtuOTgL3_uYB4wzbbBpEBmlb1o2ks5sBW99gaJpZM4NweCt> > . >

[srowen]

I think the question is whether we can get this change into the 2.2 release. It'd be ideal, but not sure it's worth another RC. Of course, this runs into the discussion we're currently having about when to update docs.

[dongjoon-hyun]

I hope it goes to Release Note instead of blocking the current voting.
Something like 2.1.0 Release Note.

[srowen]

Merged to master/2.2

[dongjoon-hyun]

Thank you all!