[SPARK-27178][k8s] add nss to the spark/k8s Dockerfile by shaneknapp · Pull Request #24111 · apache/spark

shaneknapp · 2019-03-15T22:35:23Z

What changes were proposed in this pull request?

while performing some tests on our existing minikube and k8s infrastructure, i noticed that the integration tests were failing. i dug in and discovered the following message buried at the end of the stacktrace:

 Caused by: java.io.FileNotFoundException: /usr/lib/libnss3.so
at sun.security.pkcs11.Secmod.initialize(Secmod.java:193)
at sun.security.pkcs11.SunPKCS11.<init>(SunPKCS11.java:218)
... 81 more

after i added the nss package to resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile, everything worked.

this is also impacting current builds. see: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8959/console

How was this patch tested?

i tested locally before pushing, and the build system will test the rest.

SparkQA · 2019-03-15T22:38:30Z

Test build #103556 has started for PR 24111 at commit 48e7a0c.

SparkQA · 2019-03-15T22:46:37Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8967/

SparkQA · 2019-03-15T22:47:16Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8967/

shaneknapp · 2019-03-15T22:49:06Z

test this please

SparkQA · 2019-03-15T23:01:42Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8969/

SparkQA · 2019-03-15T23:04:59Z

Test build #103558 has finished for PR 24111 at commit 48e7a0c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

erikerlandson · 2019-03-15T23:14:54Z

Was this introduced by some other dependency version bump?

SparkQA · 2019-03-15T23:18:31Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8969/

shaneknapp · 2019-03-15T23:23:10Z

Was this introduced by some other dependency version bump?

the only other PR merged that touched k8s after the client bump on wednesday was this one:
#23380

...and that doesn't touch anything i care about. :\

since i don't want to ruin my weekend, i will investigate further next week.

felixcheung · 2019-03-16T23:39:03Z

sun.security - is it running with a new JDK?

srowen

Seems OK to me, but I don't know much about this.

shaneknapp · 2019-03-18T02:20:19Z

sun.security - is it running with a new JDK?

i dunno... it's in the docker container, which is pulling upstream from alpine linux's dockerhub repo.

markhamstra · 2019-03-18T02:39:21Z

Seems that there is an argument to be made that the Dockerfile should lock down a much more specific version of Alpine and JDK. Using what is essentially a floating tag leaves us guessing any time these dependencies change.

shaneknapp · 2019-03-18T02:57:04Z

Seems that there is an argument to be made that the Dockerfile should lock down a much more specific version of Alpine and JDK. Using what is essentially a floating tag leaves us guessing any time these dependencies change.

yep, i agree completely. it's not at all where we want to be.

three things i will do tomorrow:

open a jira to discuss locking any dockerfiles to a specific version (maybe host our own?)
open a new PR for 2.4.x w/this fix
looking in to any potential dep changes in the image

sorry, four thing:

pick up a bottle of bourbon to help with (3)

:)

attilapiros · 2019-03-18T17:17:05Z

ping @vanzin

vanzin · 2019-03-18T17:48:01Z

Actually I wonder if this has been fixed already upstream. Since both the jdk and the base libs come from the base image, it would be a bug in the base image if this library were missing.

With the latest openjdk:8-alpine:

$ docker run -it openjdk:8-alpine
/ # java -version
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (IcedTea 3.10.0) (Alpine 8.191.12-r0)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
/ # ls -l /usr/lib/libnss3.so
lrwxrwxrwx 1 root root 13 Mar 8 02:13 /usr/lib/libnss3.so -> libnss3.so.41

Not sure if the IT scripts are forcing the image to refresh or use a cached version, though...

shaneknapp · 2019-03-18T18:12:57Z

Actually I wonder if this has been fixed already upstream. Since both the jdk and the base libs come from the base image, it would be a bug in the base image if this library were missing.
With the latest openjdk:8-alpine:
$ docker run -it openjdk:8-alpine
/ # java -version
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (IcedTea 3.10.0) (Alpine 8.191.12-r0)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
/ # ls -l /usr/lib/libnss3.so
lrwxrwxrwx 1 root root 13 Mar 8 02:13 /usr/lib/libnss3.so -> libnss3.so.41
Not sure if the IT scripts are forcing the image to refresh or use a cached version, though...

hmm, lemme remove the nss package install and re-run the k8s integration tests locally and see what happens.

shaneknapp · 2019-03-18T19:14:51Z

ok... so the last time this image on dockerhub was updated was march 1st, which doesn't fit in to our timing of these failures happening. i'm still mildly confused.

i'm also wondering if this is something that's minikube-specific... since we're using the convoluted path of minikube -> kvm2 -> docker-machine things may not be behaving as expected.

also, i re-tested the integration tests w/nss removed from apk add and the jobs are failing "as expected" and passing w/nss added to the Dockerfile.

vanzin · 2019-03-18T19:35:36Z

that's weird because "apk info nss" on the bare image shows that it is installed, so this change should be a no op.

$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
openjdk 8-alpine e9ea51023687 10 days ago 105MB

vanzin · 2019-03-18T19:44:54Z

Ah, I think this is it:

 ---> Running in 23e86476ab1c
+ apk upgrade --no-cache
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
(1/7) Upgrading libcrypto1.1 (1.1.1a-r1 -> 1.1.1b-r1)
(2/7) Upgrading libssl1.1 (1.1.1a-r1 -> 1.1.1b-r1)
(3/7) Upgrading openjdk8-jre-base (8.191.12-r0 -> 8.201.08-r0)
(4/7) Upgrading openjdk8-jre (8.191.12-r0 -> 8.201.08-r0)
(5/7) Purging nss (3.41-r0)

Now why the upgrade is purging the nss package, I have no idea... seems like a bug in some alpine package (probably the updated openjre 8.201.08-r0), but this seems ok as a workaround.

shaneknapp · 2019-03-18T20:17:25Z

ok cool. i'll test this one more time and then merge it to master. 2.4.1 will be it's own PR.

shaneknapp · 2019-03-18T20:17:33Z

test this please

SparkQA · 2019-03-18T20:29:19Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/9034/

SparkQA · 2019-03-18T20:34:01Z

Test build #103633 has finished for PR 24111 at commit 48e7a0c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-03-18T20:40:51Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/9034/

shaneknapp · 2019-03-18T21:09:44Z

alright, merging to master.

shaneknapp · 2019-03-18T21:16:38Z

actually i'm going to wait for a couple of hours just to be sure my access to repos is properly synced.

## What changes were proposed in this pull request? see also: #24111 while performing some tests on our existing minikube and k8s infrastructure, i noticed that the integration tests were failing. i dug in and discovered the following message buried at the end of the stacktrace: ``` Caused by: java.io.FileNotFoundException: /usr/lib/libnss3.so at sun.security.pkcs11.Secmod.initialize(Secmod.java:193) at sun.security.pkcs11.SunPKCS11.<init>(SunPKCS11.java:218) ... 81 more ``` after i added the `nss` package to `resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile`, everything worked. this is also impacting current builds. see: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8959/console ## How was this patch tested? i tested locally before pushing, and the build system will test the rest. Closes#24137 from shaneknapp/add-nss-package. Authored-by: shane knapp <incomplete@gmail.com> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>

see also: apache#24111 while performing some tests on our existing minikube and k8s infrastructure, i noticed that the integration tests were failing. i dug in and discovered the following message buried at the end of the stacktrace: ``` Caused by: java.io.FileNotFoundException: /usr/lib/libnss3.so at sun.security.pkcs11.Secmod.initialize(Secmod.java:193) at sun.security.pkcs11.SunPKCS11.<init>(SunPKCS11.java:218) ... 81 more ``` after i added the `nss` package to `resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile`, everything worked. this is also impacting current builds. see: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8959/console i tested locally before pushing, and the build system will test the rest. Closesapache#24137 from shaneknapp/add-nss-package. Authored-by: shane knapp <incomplete@gmail.com> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>

## What changes were proposed in this pull request? see also: apache#24111 while performing some tests on our existing minikube and k8s infrastructure, i noticed that the integration tests were failing. i dug in and discovered the following message buried at the end of the stacktrace: ``` Caused by: java.io.FileNotFoundException: /usr/lib/libnss3.so at sun.security.pkcs11.Secmod.initialize(Secmod.java:193) at sun.security.pkcs11.SunPKCS11.<init>(SunPKCS11.java:218) ... 81 more ``` after i added the `nss` package to `resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile`, everything worked. this is also impacting current builds. see: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8959/console ## How was this patch tested? i tested locally before pushing, and the build system will test the rest. Closesapache#24137 from shaneknapp/add-nss-package. Authored-by: shane knapp <incomplete@gmail.com> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>

## What changes were proposed in this pull request? see also: apache/spark#24111 while performing some tests on our existing minikube and k8s infrastructure, i noticed that the integration tests were failing. i dug in and discovered the following message buried at the end of the stacktrace: ``` Caused by: java.io.FileNotFoundException: /usr/lib/libnss3.so at sun.security.pkcs11.Secmod.initialize(Secmod.java:193) at sun.security.pkcs11.SunPKCS11.<init>(SunPKCS11.java:218) ... 81 more ``` after i added the `nss` package to `resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile`, everything worked. this is also impacting current builds. see: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8959/console ## How was this patch tested? i tested locally before pushing, and the build system will test the rest. Closes #24137 from shaneknapp/add-nss-package. Authored-by: shane knapp <incomplete@gmail.com> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com> (cherry picked from commit 342e91f)

adding nss to the Dockerfile
48e7a0c

srowen reviewed Mar 17, 2019
View reviewed changes

shaneknapp mentioned this pull request Mar 18, 2019
[SPARK-27178][k8s][branch-2.4] adding nss package to fix tests #24137
Closed

asfgit closed this in 5564fe5Mar 18, 2019

shaneknapp deleted the add-nss-package-to-dockerfile branch April 19, 2019 17:35

Uh oh!

Conversation

shaneknapp commented Mar 15, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Mar 15, 2019

Uh oh!

SparkQA commented Mar 15, 2019

Uh oh!

SparkQA commented Mar 15, 2019

Uh oh!

shaneknapp commented Mar 15, 2019

Uh oh!

SparkQA commented Mar 15, 2019

Uh oh!

SparkQA commented Mar 15, 2019

Uh oh!

erikerlandson commented Mar 15, 2019

Uh oh!

SparkQA commented Mar 15, 2019

Uh oh!

shaneknapp commented Mar 15, 2019

Uh oh!

felixcheung commented Mar 16, 2019

Uh oh!

srowen left a comment

Choose a reason for hiding this comment

Uh oh!

shaneknapp commented Mar 18, 2019

Uh oh!

markhamstra commented Mar 18, 2019

Uh oh!

shaneknapp commented Mar 18, 2019

Uh oh!

attilapiros commented Mar 18, 2019

Uh oh!

vanzin commented Mar 18, 2019

Uh oh!

shaneknapp commented Mar 18, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shaneknapp commented Mar 18, 2019

Uh oh!

vanzin commented Mar 18, 2019

Uh oh!

vanzin commented Mar 18, 2019

Uh oh!

shaneknapp commented Mar 18, 2019

Uh oh!

shaneknapp commented Mar 18, 2019

Uh oh!

SparkQA commented Mar 18, 2019

Uh oh!

SparkQA commented Mar 18, 2019

Uh oh!

SparkQA commented Mar 18, 2019

Uh oh!

shaneknapp commented Mar 18, 2019

Uh oh!

shaneknapp commented Mar 18, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

shaneknapp commented Mar 15, 2019•
edited
Loading

shaneknapp commented Mar 18, 2019•
edited
Loading