[SPARK-14204] [SQL] register driverClass rather than user-specified class by mchalek · Pull Request #12000 · apache/spark

mchalek · 2016-03-28T16:26:43Z

This pull request fixes an issue in which cluster-mode executors fail to properly register a JDBC driver when the driver is provided in a jar by the user, but the driver class name is derived from a JDBC URL (rather than specified by the user). The consequence of this is that all JDBC accesses under the described circumstances fail with an IllegalStateException. I reported the issue here: https://issues.apache.org/jira/browse/SPARK-14204

My proposed solution is to have the executors register the JDBC driver class under all circumstances, not only when the driver is specified by the user.

This patch was tested manually. I built an assembly jar, deployed it to a cluster, and confirmed that the problem was fixed.

holdenk · 2016-03-29T16:52:20Z

That seems like a pretty reasonable change, although it might be good to add a regression test. Maybe @davies or @JoshRosen can verify this patch for Jenkins?

mchalek · 2016-03-29T19:27:42Z

Thanks @holdenk I agree about the value of a regression test for this. I'm not familiar with how Spark's regression testing works. Is there a mechanism through which to run tests that emulate cluster-mode spark, with isolated master/executor JVMs, as well as isolated primordial and task JVMs for the executor? I think it may be necessary to have all 3 JVMs isolated in order to test this issue.

holdenk · 2016-03-29T19:32:49Z

There are some integration tests hosted in an separate repo by the Databricks people at https://github.com/databricks/spark-integration-tests , I'm not super sure on the policy for contributing to that but maybe one the Databricks people can chime in (or we could just leave doing that as a follow up activity).

mchalek · 2016-04-05T14:48:40Z

Bump. Would be nice to get closure on this. I doubt that we were alone in being affected by this (although admittedly, complaints of other failures seem not to have yet surfaced).

srowen · 2016-05-04T15:21:14Z

@JoshRosen do you have a thought on this since you made the change?

zzcclp · 2016-05-17T05:54:15Z

@mchalek @JoshRosen Is there any progress yet? I got the same error with spark-1.6.1. But if I add "driver=com.mysql.jdbc.Driver" property to write.jdbc's properties params, i was ok.
This pr looks like a pretty reasonable change without setting "driver=com.mysql.jdbc.Driver" property.

JoshRosen · 2016-05-18T00:09:11Z

I'm on vacation right now. This patch seems pretty small and reasonable to me, though, so it would be great if someone else could take a final look and merge if appropriate.

zzcclp · 2016-05-26T03:33:58Z

@JoshRosen , could you merge this into branch-1.6?

zzcclp · 2016-05-30T02:05:34Z

ping @JoshRosen

zzcclp · 2016-06-01T22:03:02Z

@srowen @JoshRosen @holdenk could you take a look and merge this pr?

srowen · 2016-06-02T02:41:17Z

Jenkins test this please

SparkQA · 2016-06-02T02:49:30Z

Test build #59804 has finished for PR 12000 at commit a927c76.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2016-06-02T03:05:05Z

Jenkins retest this please

SparkQA · 2016-06-02T04:34:54Z

Test build #59806 has finished for PR 12000 at commit a927c76.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

This pull request fixes an issue in which cluster-mode executors fail to properly register a JDBC driver when the driver is provided in a jar by the user, but the driver class name is derived from a JDBC URL (rather than specified by the user). The consequence of this is that all JDBC accesses under the described circumstances fail with an `IllegalStateException`. I reported the issue here: https://issues.apache.org/jira/browse/SPARK-14204 My proposed solution is to have the executors register the JDBC driver class under all circumstances, not only when the driver is specified by the user. This patch was tested manually. I built an assembly jar, deployed it to a cluster, and confirmed that the problem was fixed. Author: Kevin McHale <kevin@premise.com> Closes#12000 from mchalek/jdbc-driver-registration.

srowen · 2016-06-02T16:18:35Z

Merged to 1.6 per Josh

mchalek · 2016-06-02T16:33:30Z

Cool, thanks for pushing on this @zzcclp and for merging it @srowen. Any idea when 1.6.2 will be released?

This pull request fixes an issue in which cluster-mode executors fail to properly register a JDBC driver when the driver is provided in a jar by the user, but the driver class name is derived from a JDBC URL (rather than specified by the user). The consequence of this is that all JDBC accesses under the described circumstances fail with an `IllegalStateException`. I reported the issue here: https://issues.apache.org/jira/browse/SPARK-14204 My proposed solution is to have the executors register the JDBC driver class under all circumstances, not only when the driver is specified by the user. This patch was tested manually. I built an assembly jar, deployed it to a cluster, and confirmed that the problem was fixed. Author: Kevin McHale <kevin@premise.com> Closesapache#12000 from mchalek/jdbc-driver-registration. (cherry picked from commit 0a13e4c)

AmplabJenkins · 2016-07-04T04:42:14Z

Can one of the admins verify this patch?

zzcclp · 2016-07-27T07:29:02Z

@srowen @JoshRosen does this pr need to be merged into master or branch-2.0 ?

srowen · 2016-07-27T07:51:59Z

I had the impression this was specific to the 1.x code, and not an issue in 2.x?

mchalek · 2016-07-27T19:13:11Z

The problem is with commit 7f37c1e, so if that commit is on master then this needs to be merged into master as well. Taking a quick look at the affected file, it does appear that the incorrect code is in master.

mchalek · 2016-07-27T19:15:54Z

Ah, the hash for the relevant commit on master is 6c83d93

zzcclp · 2016-07-29T09:32:56Z

Hi, @srowen @JoshRosen @mchalek , this issue still exists on branch-2.0/master, if I add "driver=com.mysql.jdbc.Driver" property to write.jdbc's properties params, it was ok, otherwise failed.

srowen · 2016-07-29T12:05:24Z

OK open a new PR for master

zzcclp · 2016-07-30T06:14:48Z

@srowen @mchalek could you open a new pr for master/branch-2.0?

zzcclp · 2016-07-30T14:49:07Z

thanks, @mchalek

This is a pull request that was originally merged against branch-1.6 as #12000, now being merged into master as well. srowen zzcclp JoshRosen This pull request fixes an issue in which cluster-mode executors fail to properly register a JDBC driver when the driver is provided in a jar by the user, but the driver class name is derived from a JDBC URL (rather than specified by the user). The consequence of this is that all JDBC accesses under the described circumstances fail with an IllegalStateException. I reported the issue here: https://issues.apache.org/jira/browse/SPARK-14204 My proposed solution is to have the executors register the JDBC driver class under all circumstances, not only when the driver is specified by the user. This patch was tested manually. I built an assembly jar, deployed it to a cluster, and confirmed that the problem was fixed. Author: Kevin McHale <kevin@premise.com> Closes#14420 from mchalek/mchalek-jdbc_driver_registration. (cherry picked from commit 685b08e) Signed-off-by: Sean Owen <sowen@cloudera.com>

srowen · 2016-08-03T20:16:28Z

PS @mchalek can you manually close this PR? it's merged but can't be auto closed

This is a pull request that was originally merged against branch-1.6 as #12000, now being merged into master as well. srowen zzcclp JoshRosen This pull request fixes an issue in which cluster-mode executors fail to properly register a JDBC driver when the driver is provided in a jar by the user, but the driver class name is derived from a JDBC URL (rather than specified by the user). The consequence of this is that all JDBC accesses under the described circumstances fail with an IllegalStateException. I reported the issue here: https://issues.apache.org/jira/browse/SPARK-14204 My proposed solution is to have the executors register the JDBC driver class under all circumstances, not only when the driver is specified by the user. This patch was tested manually. I built an assembly jar, deployed it to a cluster, and confirmed that the problem was fixed. Author: Kevin McHale <kevin@premise.com> Closes#14420 from mchalek/mchalek-jdbc_driver_registration.

register driverClass rather than user class
a927c76

mchalek changed the title ~~[SPARK-14204] [SQL] register driverClass rather than user class~~[SPARK-14204] [SQL] register driverClass rather than user-specified classMar 28, 2016

mchalek mentioned this pull request Jul 30, 2016
[SPARK-14204] [SQL] register driverClass rather than user-specified class #14420
Closed

srowen mentioned this pull request Aug 4, 2016
MAINTENANCE. Cleaning up stale PRs. #14495
Closed

asfgit closed this in 53e766cAug 4, 2016

Uh oh!

Conversation

mchalek commented Mar 28, 2016

Uh oh!

holdenk commented Mar 29, 2016

Uh oh!

mchalek commented Mar 29, 2016

Uh oh!

holdenk commented Mar 29, 2016

Uh oh!

mchalek commented Apr 5, 2016

Uh oh!

srowen commented May 4, 2016

Uh oh!

zzcclp commented May 17, 2016

Uh oh!

JoshRosen commented May 18, 2016

Uh oh!

zzcclp commented May 26, 2016

Uh oh!

zzcclp commented May 30, 2016

Uh oh!

zzcclp commented Jun 1, 2016

Uh oh!

srowen commented Jun 2, 2016

Uh oh!

SparkQA commented Jun 2, 2016

Uh oh!

srowen commented Jun 2, 2016

Uh oh!

SparkQA commented Jun 2, 2016

Uh oh!

srowen commented Jun 2, 2016

Uh oh!

mchalek commented Jun 2, 2016

Uh oh!

AmplabJenkins commented Jul 4, 2016

Uh oh!

zzcclp commented Jul 27, 2016

Uh oh!

srowen commented Jul 27, 2016

Uh oh!

mchalek commented Jul 27, 2016

Uh oh!

mchalek commented Jul 27, 2016

Uh oh!

zzcclp commented Jul 29, 2016

Uh oh!

srowen commented Jul 29, 2016

Uh oh!

zzcclp commented Jul 30, 2016

Uh oh!

zzcclp commented Jul 30, 2016

Uh oh!

srowen commented Aug 3, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants