[do not merge] debug seccomp-related issues on armhf (arm32v7)

[thaJeztah]

Carrying #142, because Jenkins doesn't run with updated changes unless the author has write access

[thaJeztah]

Hm.. failure on arm64;

#8 30.27 Unpacking libc6:armhf (2.31-0ubuntu6) over (2.30-0ubuntu3) ...
#8 42.35 tar: ./control: Cannot utime: Operation not permitted
#8 42.35 tar: ./md5sums: Cannot utime: Operation not permitted
#8 42.35 tar: ./shlibs: Cannot utime: Operation not permitted
#8 42.35 tar: ./symbols: Cannot utime: Operation not permitted
#8 42.35 tar: ./triggers: Cannot utime: Operation not permitted
#8 42.35 tar: .: Cannot utime: Operation not permitted
#8 42.35 tar: Exiting with failure status due to previous errors
#8 42.35 dpkg-deb: error: tar subprocess returned error exit status 2
#8 42.35 dpkg: error processing archive /var/cache/apt/archives/libcrypt1_1%3a4.4.10-10ubuntu4_armhf.deb (--unpack):
#8 42.35 dpkg-deb --control subprocess returned error exit status 2
#8 42.37 Errors were encountered while processing:
#8 42.37 /var/cache/apt/archives/libcrypt1_1%3a4.4.10-10ubuntu4_armhf.deb
#8 42.53 E: Sub-process /usr/bin/dpkg returned an error code (1)
#8 ERROR: executor failed running [/bin/sh -c apt-get update && apt-get install -y --no-install-recommends curl devscripts equivs git lsb-release && apt-get clean && rm -rf /var/lib/apt/lists/*]: runc did not terminate sucessfully
------
> [stage-5 3/14] RUN apt-get update && apt-get install -y --no-install-recommends curl devscripts equivs git lsb-release && apt-get clean && rm -rf /var/lib/apt/lists/*:
------
failed to solve with frontend dockerfile.v0: failed to build LLB: executor failed running [/bin/sh -c apt-get update && apt-get install -y --no-install-recommends curl devscripts equivs git lsb-release && apt-get clean && rm -rf /var/lib/apt/lists/*]: runc did not terminate sucessfully
Makefile:64: recipe for target 'build' failed
make: *** [build] Error 1

[DeeDeeG]

Something to do with insufficient permissions when running tar? https://superuser.com/questions/1219214/permissions-cannot-be-restored-for-a-tar

(Though everything is generally run as root in Docker, so... trying to write to a read-only or locked filesystem/directory??)

[thaJeztah]

Yeah, thinking either (e.g.) seccomp blocking something, or perhaps an issue with overlayfs or the backing filesystem.

Let me push a commit to debug docker info, docker version, and run the check-config script

[thaJeztah]

Found someone else running into this; dotnet/dotnet-docker#1747

It's not clear whether https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1867431 is related. > So I've logged https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1867675 for this issue.

[thaJeztah]

Checking if seccomp is possibly blocking it; moved the failing step to the docker run (instead of doing during docker build, and disabled seccomp (--security-opt seccomp=unconfined)

[DeeDeeG]

If not seccomp-related, this tracks pretty closely with what people are reporting in: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1867431

This was a recent gnarly packaging issue where the absolute path of a shared object (.so) file got lost in translation... Two packages thought it was in two different places. So, attempting to write/rm said shared object during package upgrades fails. Or something like that.

Per this comment:

After you've hit this problem, installing/reinstalling the *new* version of libcrypt1 >(1:4.4.10-10ubuntu4) should be enough to correct the failure.

Edit to add, from this later comment:

if you didn't yet upgrade from an earlier focal version, make sure that you remove the libcrypt1 package before the upgrade. Then do a normal upgrade.

[thaJeztah]

Thanks; yes, I was glancing over that issue, and would be plausible as well. Just pushed again, because I forgot to set DEBIAN_FRONTEND=noninteractive, causing installation getting stuck because tzdata was waiting for input (sigh) 😞

[thaJeztah]

OK, so the good news is that with seccomp disabled it works (so at least narrowed down where to look for). The "bad" news is that we now need to figure out what's blocked, and if installing the package is making different syscalls (and if those should be either "not blocked", or if the package needs changing)

[thaJeztah]

For posterity; the failure occurred on; a 4.4.127-mainline-rev1 kernel, Ubuntu 16.04.5 LTS;

Client:
Debug Mode: false
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 4
Server Version: 19.03.5
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.127-mainline-rev1
Operating System: Ubuntu 16.04.5 LTS
OSType: linux
Architecture: armv7l
CPUs: 4
Total Memory: 1.974GiB
Name: arm32v7-ubuntu-13
ID: STIC:WY64:VMVN:7XQN:QPG7:HL2J:XZDK:VN7J:L4UG:WJEY:ZV3E:MK3X
Docker Root Dir: /var/lib/docker
Debug Mode: false
Username: dockerbuildbot
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

[DeeDeeG]

I'm unfamiliar with seccomp, TBH, but does something like this fix it?

apt update \
&& apt install --reinstall libcrypt1 \
&& apt install [other packages]

ETA: (Maybe trying to delete a file that's not actually in the package runs afoul of seccomp?)

[thaJeztah]

I'm unfamiliar with seccomp, TBH

The seccomp feature uses a whitelist of syscalls that a process (container in this case) can make. Any syscall that is not whitelisted will be blocked (thus making the container more secure).

Chatting with a colleague, it's possible that new syscalls were added for 32 bit platforms that are not yet included in either libseccomp or in the default profile that docker uses (therefore blocking those new syscalls).

[thaJeztah]

Pushed a commit to check if reinstalling libcrypt1 works (combined with re-enabling seccomp)

[thaJeztah]

Reinstall libcrypt1 does not help, so it's really seccomp or the package to look into

[DeeDeeG]

Not sure if this is helpful, but there are things here that sound related. From the changelog for libc6 in Focal:

glibc (2.31-0ubuntu1) focal; urgency=medium

• Merge with current Debian git glibc-2.31.
• debian/patches/git-updates.diff: update from upstream stable branch.
• Ignore test failures for sysvipc/test-sysvmsg, sysvipc/test-sysvsem and
  sysvipc/test-sysvshm on 32bit architectures, failing on the xenial kernel,
  succeeding on the bionic and focal kernels.
• Restore the __glibc_has_include macro, needed until GCC is rebuilt
  to not include this in the fixed-include headers.
• Backport 5828bc4523230685ac29a4a882967913255f5666, making the clone3
  syscall known on arm64, fixing misc/tst-glibcsyscalls.
• Ignore some float tests for the non-default armel multilib variant.
  https://sourceware.org/ml/libc-alpha/2020-03/msg00074.html

-- Matthias Klose <doko [at] ubuntu.com> Fri, 06 Mar 2020 12:06:42 +0100

Particularly the point about changing syscalls on ARM.

• Backport 5828bc4523230685ac29a4a882967913255f5666, making the clone3
  syscall known on arm64, fixing misc/tst-glibcsyscalls.

[DeeDeeG]

One last thought: If your CI's base image can be updated to a more recent image of focal, with recent libc6, that would get this apt install step out of the way, and I suppose CI wouldn't fail on this (seemingly unrelated) PR.

When I check in the latest focal image, libc6 seems to be the version your CI is trying to upgrade to:

$ docker run ubuntu:focal apt install libc6
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Reading package lists...
Building dependency tree...
Reading state information...
libc6 is already the newest version (2.31-0ubuntu6).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Though admittedly I am doing this with an amd64 host and image, so maybe this problem isn't solved in the latest ARM images/maybe the ARM images failed to update due to this issue?

[thaJeztah]

Gonna use this PR for debugging the issue only, and opened #152 to add Ubuntu 20.04, but skipping arm32 for now

[thaJeztah]

Unfortunately, looks like the updated seccomp profile does not address the issue;

Unpacking libc6:armhf (2.31-0ubuntu6) over (2.30-0ubuntu3) ...
tar: ./control: Cannot utime: Operation not permitted
tar: ./md5sums: Cannot utime: Operation not permitted
tar: ./shlibs: Cannot utime: Operation not permitted
tar: ./symbols: Cannot utime: Operation not permitted
tar: ./triggers: Cannot utime: Operation not permitted
tar: .: Cannot utime: Operation not permitted
tar: Exiting with failure status due to previous errors

libseccomp version installed:

apt list libseccomp2 -a
libseccomp2/xenial-updates,xenial-security,now 2.4.1-0ubuntu0.16.04.2 armhf [installed]
libseccomp2/xenial 2.2.3-3ubuntu3 armhf

[thaJeztah]

ok; "success" - the problem is solved when installing libseccomp 2.4.3. Unfortunately, that version is not available on Ubuntu versions < 20.03 (https://packages.ubuntu.com/search?keywords=libseccomp2).

So for debugging, I installed the package from the ubuntu 20.03 repository.

What it comes down to;

The container we're running (ubuntu:20.03) makes a syscall that's introduced in Linux 5.x, but docker in this case is running on a 4.x kernel (the host is Ubuntu 16.04). The version of libseccomp installed on the host is not taking kernel 5.x syscalls into account, receives an error, and (likely) in that case blocks the syscall.

Solutions for this would be to;

• ask Ubuntu and Debian package maintainers to provide libseccomp 2.4.3 packages for older (LTS) releases. It's a patch release, so possibly acceptable for them. On the other hand; it's adding "features" for a kernel version that's not used by those versions of Ubuntu / Debian.
• somehow make libseccomp handle "unknown" syscalls, and perhaps allow them (instead of blocking)? (not exactly sure how it's handling these, so I'd have to read up on that); probably that's the same (similar) as changing our "whitelist" to a "blacklist" (which could weaken security)

[mthalman]

@thaJeztah - Have you managed to resolve this now that an updated version of libseccomp has been released? My Docker host machine is running Ubuntu Xenial and it has libseccomp (2.4.3-1ubuntu3.16.04.2) installed but is still running into this Cannot utime: Operation not permitted error. I'm wondering if it's because my machine has AArch64 architecture.