[SPARK-27168][SQL][TEST] Add docker integration test for MsSql server

[zhulipeng]

What changes were proposed in this pull request?

This PR aims to add a JDBC integration test for MsSql server.

How was this patch tested?

./build/mvn clean install -DskipTests
./build/mvn test -Pdocker-integration-tests -pl :spark-docker-integration-tests_2.12 \
-Dtest=none -DwildcardSuites=org.apache.spark.sql.jdbc.MsSqlServerIntegrationSuite

[srowen]

I am not sure how much SQL Server is intended to be supported or how much overhead this testing adds. I'm not against it.

[zhulipeng]

@srowen These docker integration tests are in external directory, they need user to manually kickoff in locally, I think it will not affects the build process overheads.

[dongjoon-hyun]

ok to test

[dongjoon-hyun]

@lipzhu . This test suite should be verified manually. And, I cannot pass the tests with AS-IS PR code. @srowen . Was this okay in your environment? I want to be sure about that.

[dongjoon-hyun]

Oh, I got it. We need the previous 8ee09f2 on Friday. Let me rebase this PR and test again.
Hmm. After rebasing and building cleanly, I'm still facing the same failure.

[zhulipeng]

Oh, I got it. We need the previous 8ee09f2 on Friday. Let me rebase this PR and test again.
Hmm. After rebasing and building cleanly, I'm still facing the same failure.

@dongjoon-hyun I just double confirm in local, the MsSqlServerIntegrationSuite can success.
Can you paste the error log here?

BTW, The memory setting have some gaps between tag latest and 2017-GA-ubuntu
Below is the memory requirement for docker container.

[SparkQA]

Test build #103595 has finished for PR 24099 at commit 7a844e1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #103598 has finished for PR 24099 at commit 1b4bd43.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[zhulipeng]

reset this please.

[dongjoon-hyun]

Interesting. The error log was #24099 (comment) .
And, my Docker machine configuration is 10GB RAM / 8 Core / DockerDesktop Community(2.0.0.3) / Mac.

I'll try again since this is updated. Thanks, @lipzhu .

[dongjoon-hyun]

@lipzhu . If you clear your maven repository and run ./build/mvn test -Pdocker-integration-tests -pl :spark-docker-integration-tests_2.12, you can see the failure.

For Apache Spark contribution, I'd like to recommend you the followings.

• To get a correct review, the PR itself should be a complete code, but this is not.
  Although Jenkins builds PRs after rebasing, please don't assume that the reviewer to rebase your PR to the master branch. Technically, we need 8ee09f2 in this PR, isn't it?

• Before testing locally, please remove your local maven repository. It's an essential step for sanity. Docker integration suite uses the snapshot artifacts in the following location. However, your previous PR is not published there yet.

  • https://repository.apache.org/content/groups/snapshots/org/apache/spark/spark-sql_2.12/3.0.0-SNAPSHOT/

Due to (1) and (2), only you pass the test locally.

Technically, this PR needs three following steps. Otherwise, it fails.

1. Rebased this PR to bring the previous patch.
2. Do mvn clean -DskipTests install.
3. Run build/mvn test -Pdocker-integration-tests -pl :spark-docker-integration-tests_2.12.

Lastly, please update How was this patch tested? section in this PR description. Otherwise, we need to wait until the snapshot artifacts are ready.

[zhulipeng]

Got it, thanks for your comments and review. @dongjoon-hyun

[dongjoon-hyun]

Thank you for updating, @lipzhu ! I'll review again in a few hours.

[dongjoon-hyun]

Hi, @lipzhu .
I updated the PR description and made a refactoring PR to you. Could you review and merge that please? (I know this PR follows the existing style, but the existing style is a little old for Spark 3.0.)

• https://github.com/lipzhu/spark/pull/1

[dongjoon-hyun]

Thank you for review and applying.

[dongjoon-hyun]

+1, LGTM (Pending Jenkins).
Thank you for doing this.

[SparkQA]

Test build #103655 has finished for PR 24099 at commit 65f6fe8.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• assert(types(1).equals(\"class java.lang.Integer\"))
• assert(types(2).equals(\"class java.lang.Integer\"))
• assert(types(3).equals(\"class java.lang.Integer\"))
• assert(types(4).equals(\"class java.lang.Long\"))
• assert(types(5).equals(\"class java.lang.Double\"))
• assert(types(6).equals(\"class java.lang.Double\"))
• assert(types(7).equals(\"class java.lang.Double\"))
• assert(types(8).equals(\"class java.math.BigDecimal\"))
• assert(types(11).equals(\"class java.math.BigDecimal\"))

[SparkQA]

Test build #103649 has finished for PR 24099 at commit bc6ed59.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[dilipbiswal]

retest this please

[SparkQA]

Test build #103665 has finished for PR 24099 at commit 65f6fe8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• assert(types(1).equals(\"class java.lang.Integer\"))
• assert(types(2).equals(\"class java.lang.Integer\"))
• assert(types(3).equals(\"class java.lang.Integer\"))
• assert(types(4).equals(\"class java.lang.Long\"))
• assert(types(5).equals(\"class java.lang.Double\"))
• assert(types(6).equals(\"class java.lang.Double\"))
• assert(types(7).equals(\"class java.lang.Double\"))
• assert(types(8).equals(\"class java.math.BigDecimal\"))
• assert(types(11).equals(\"class java.math.BigDecimal\"))

[dongjoon-hyun]

Thank you, @lipzhu , @srowen , @dilipbiswal .
Merged to master.

[dongjoon-hyun]

Hi, All (and @gatorsmile )
I'll backport this test patch from master to branch-2.4 in order to support Apache Spark 2.4.x LTS branch in a more robust way.

Since this integration test suite is not tested on Jenkins, I cherry-picked and tested locally.
(Backporting was a clean cherry-pick.)