From d6c59a7d14abe81a4d787100c83a32b75c38aed1 Mon Sep 17 00:00:00 2001 From: Kent Yao Date: Mon, 22 Jan 2024 13:45:40 +0800 Subject: [PATCH 1/2] [SPARK-46788][DOCS] Add instructions for running Docker integration tests --- connector/docker-integration-tests/README.md | 111 +++++++++++++++++++ 1 file changed, 111 insertions(+) create mode 100644 connector/docker-integration-tests/README.md diff --git a/connector/docker-integration-tests/README.md b/connector/docker-integration-tests/README.md new file mode 100644 index 0000000000000..1d1ac21874440 --- /dev/null +++ b/connector/docker-integration-tests/README.md @@ -0,0 +1,111 @@ +--- +layout: global +title: Spark Docker Integration Tests +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +# Running the Docker integration tests + +Note that the integration test framework is using thirdparty Docker images to test functionalities +of integration with other data sources, not running Spark itself in Docker containers, which is +actually done by the Kubernetes Integration Tests module. + +In order to run the Docker integration tests, the [Docker engine](https://docs.docker.com/engine/installation/) +needs to be installed and started on the machine. Additionally, the environment variable +`ENABLE_DOCKER_INTEGRATION_TESTS=1` shall be specified to enable the Docker integration tests. + + export ENABLE_DOCKER_INTEGRATION_TESTS=1 + +or, + + ENABLE_DOCKER_INTEGRATION_TESTS=1 ./build/sbt -Pdocker-integration-tests "docker-integration-tests/test" + +## Running an individual Docker integration test + +Testing the whole module of Docker Integration Tests might be time-consuming because of image pulling and +the container bootstrapping. To run an individual Docker integration test, use the following command: + + ./build/sbt -Pdocker-integration-tests "docker-integration-tests/testOnly " + +## Using a custom Docker image + +Besides the default Docker images, the integration tests can be run with custom Docker images. For example, + + ORACLE_DOCKER_IMAGE_NAME=gvenzl/oracle-free:23.3-slim-faststart ./build/sbt -Pdocker-integration-tests "docker-integration-tests/testOnly *OracleIntegrationSuite" + +The following environment variables can be used to specify the custom Docker images for different databases: + +- DB2_DOCKER_IMAGE_NAME +- MARIADB_DOCKER_IMAGE_NAME +- MSSQLSERVER_DOCKER_IMAGE_NAME +- MYSQL_DOCKER_IMAGE_NAME +- ORACLE_DOCKER_IMAGE_NAME +- POSTGRES_DOCKER_IMAGE_NAME + +## Using a custom Docker context + +In certain scenarios, you may want to use a different docker context/endpoint instead of the default provided by +Docker Desktop. + + docker context ls + NAME TYPE DESCRIPTION DOCKER ENDPOINT KUBERNETES ENDPOINT ORCHESTRATOR + default * moby Current DOCKER_HOST based configuration unix:///var/run/docker.sock + desktop-linux moby Docker Desktop unix:///Users/hzyaoqin/.docker/run/docker.sock + + docker context use desktop-linux + +Then you can run the integration tests as usual targeting the Docker endpoint named `desktop-linux`. + +This is useful when the upstream Docker image can not starton the Docker Desktop. For example, when the image you +use only supports `x86_64` architecture, but you are running on an Apple Silicon `aarch64` machine, for which case +you may want to use a custom context that is able to mock a `x86_64` architecture, such as `colima`. After +[colima](https://github.com/abiosoft/colima) installed, you can start a runtime with `x86_64` support and run the +integration tests as follows: + + colima start --arch x86_64 --memory 8 --network-address + docker context colima + ./build/sbt -Pdocker-integration-tests "docker-integration-tests/testOnly *OracleIntegrationSuite" + +## Available Properties + +The following are the available properties that can be passed to optimize testing experience. + + ./build/sbt -Pdocker-integration-tests \ + -Dspark.test.docker.keepContainer=true \ + "testOnly *MariaDBKrbIntegrationSuite" + + + + + + + + + + + + + + + + + +
PropertyDescriptionDefault
spark.test.docker.keepContainer + When true, the Docker container used for the test will keep running after all tests finished in a single test file. + false
spark.test.docker.removePulledImage + When true, the Docker image used for the test will not be preserved after all tests finished in a single test file. + true
From 9a7ef029c7e33ae1d87ea931d1266bb2d6dfbdc5 Mon Sep 17 00:00:00 2001 From: Kent Yao Date: Mon, 22 Jan 2024 13:51:47 +0800 Subject: [PATCH 2/2] [SPARK-46788][DOCS] Add instructions for running Docker integration tests --- connector/docker-integration-tests/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/connector/docker-integration-tests/README.md b/connector/docker-integration-tests/README.md index 1d1ac21874440..61501be91ca21 100644 --- a/connector/docker-integration-tests/README.md +++ b/connector/docker-integration-tests/README.md @@ -62,9 +62,9 @@ In certain scenarios, you may want to use a different docker context/endpoint in Docker Desktop. docker context ls - NAME TYPE DESCRIPTION DOCKER ENDPOINT KUBERNETES ENDPOINT ORCHESTRATOR + NAME TYPE DESCRIPTION DOCKER ENDPOINT KUBERNETES ENDPOINT ORCHESTRATOR default * moby Current DOCKER_HOST based configuration unix:///var/run/docker.sock - desktop-linux moby Docker Desktop unix:///Users/hzyaoqin/.docker/run/docker.sock + desktop-linux moby Docker Desktop unix:///Users/.../docker.sock docker context use desktop-linux