Skip to content

Handle DriverWrapper when scanning registered drivers in JDBWrapper.getConnector() - #147

Closed
JoshRosen wants to merge 1 commit into
masterfrom
jdbc-driver-precedence-round-2
Closed

Handle DriverWrapper when scanning registered drivers in JDBWrapper.getConnector()#147
JoshRosen wants to merge 1 commit into
masterfrom
jdbc-driver-precedence-round-2

Conversation

@JoshRosen

Copy link
Copy Markdown
Contributor

This is a followup to #143 which fixes a corner-case bug in our scanning of registered JDBC drivers: we need to properly handle Spark's DriverWrapper drivers, which are used to wrap JDBC drivers in order to make them accessible from the root classloader so that the DriverManager can find them.

A simpler, reflection-free version of this change was incorporated into apache/spark#10519

@JoshRosenJoshRosen added this to the 0.6.0 milestone Jan 4, 2016
@codecov-io

Copy link
Copy Markdown

Current coverage is 89.18%

Merging #147 into master will not affect coverage as of 61c04da

@@ master #147 diff @@
======================================
Files 13 13 Stmts 638 638 Branches 140 140 Methods 0 0 ======================================
Hit 569 569 Partial 0 0 Missed 69 69 

Review entire Coverage Diff as of 61c04da

Powered by Codecov. Updated on successful CI builds.

@JoshRosen

Copy link
Copy Markdown
ContributorAuthor

Unfortunately, this is a little tricky to test in an automated fashion because we need to rig the classloaders in order to make sure that the test JDBC drivers aren't loaded by the root classloader.

As a pragmatic trade-off, I'm inclined to test this manually using spark-shell and --packages with the Postgres driver. I'll do this tomorrow.

@JoshRosen

Copy link
Copy Markdown
ContributorAuthor

I tested this manually with spark-shell, --jars, and a modified copy of this code with additional logging statements. I ran my manual tests on Spark 1.4.1, 1.5.1, and 2.0.0-SNAPSHOT, so I'm confident that the portions of the reflection code here which aren't covered by unit tests will work in those versions.

In case we don't get automated tests for this in place by the time of the next release, I'll open a PR to document the manual testing procedure and will create a checklist so future release coordinators / maintainers can know how to ensure that this doesn't break.

I'm going to merge this now and will begin preparation for a new spark-redshift release tomorrow.

@JoshRosen
JoshRosen deleted the jdbc-driver-precedence-round-2 branch January 6, 2016 08:41
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@JoshRosen@codecov-io