contrib/rootfs-builder: Support timestamps and xz compression

[wking]

I'm not sure when the change happened (sometime since ff5e578, #479), but Gentoo is now using timestamps and xz compression for their amd64 stage3:

$ curl http://distfiles.gentoo.org/releases/amd64/autobuilds/latest-stage3.txt
# Latest as of Fri, 09 Mar 2018 15:00:02 +0000
# ts=152060760220180308T214502Z/stage3-amd64-20180308T214502Z.tar.xz 18885107220180308T214502Z/hardened/stage3-amd64-hardened-20180308T214502Z.tar.xz 17475914020180308T214502Z/hardened/stage3-amd64-hardened+nomultilib-20180308T214502Z.tar.xz 16763913620180301T214503Z/hardened/stage3-amd64-hardened-selinux-20180301T214503Z.tar.xz 18731966420180308T214502Z/stage3-amd64-nomultilib-20180308T214502Z.tar.xz 18186629620180303/systemd/stage3-amd64-systemd-20180303.tar.bz2 28187316220180225/uclibc/stage3-amd64-uclibc-hardened-20180225.tar.bz2 5714739220180225/uclibc/stage3-amd64-uclibc-vanilla-20180225.tar.bz2 14449033220180308T214502Z/stage3-x32-20180308T214502Z.tar.xz 20528516020180308T214502Z/hardened/stage4-amd64-hardened+minimal-20180308T214502Z.tar.bz2 26177643620180308T214502Z/hardened/stage4-amd64-hardened+minimal-nomultilib-20180308T214502Z.tar.bz2 25182124520180308T214502Z/stage4-amd64-minimal-20180308T214502Z.tar.bz2 28091856520180308T214502Z/stage4-amd64-minimal-nomultilib-20180308T214502Z.tar.bz2 26883586020180303/systemd/stage4-amd64-systemd-20180303.tar.bz2 622067712

This commit adapts to that change, allowing us to continue to pull the vanilla stage3-amd64-20180308T214502Z.tar.xz.

[wking]

This needs more work for symlink and Makefile support. I'll remove the WIP prefix once I've got that sorted.

[wking]

Ok, I've cherry-picked @alban's 0453de2 and 5f15151 from #597 and added some more commits to improve get-stage3.sh's argument handling and get it working with optional stage3 timestamps (vs. the old dates) and compression (vs. the old hard-coded bz2). The rebuilt tarballs get the new directories needed for #597 (via the 5f15151 cherry-pick), and they also bump from BusyBox 1.25.1 to 1.28.0, removing catv and adding a bunch of commands listed in the a7f94a2 commit message. I think this is good to go.

[liangchenye]

It fails in my test, in the latest version, it does have commands like 'arping/su/mount/passwd/umount/' .
Is it something wrong of gentoo autobuild?

[wking]

On Mon, Mar 12, 2018 at 10:05:25AM +0000, 梁辰晔 (Liang Chenye) wrote: It fails in my test, in the latest version…

Travis is happy with it [1], although I'm not sure those tests care about the root tarballs. Can you provide details about the failure you're seeing?

… it does have commands like 'arping/su/mount/passwd/umount/' .

It has those: $ sha256sum rootfs-amd64.tar.gz 197f5faf022dd5e5d25336c5721f62b1714f9ed5c44055d6f4ab828ae11edf5d rootfs-amd64.tar.gz $ tar -tf rootfs-amd64.tar.gz | grep '/arping\|/su$\|/mount$\|/passwd\|umount' ./bin/arping ./bin/mount ./bin/passwd ./bin/su ./bin/umount ./etc/passwd [1]: https://travis-ci.org/opencontainers/runtime-tools/builds/351488881

[liangchenye]

@wking the rootfs-amd64.tar.gz works good.
I meet this issue when I 'make rootfs-amd64.tar.gz'.

ln: failed to create symbolic link 'rootfs/amd64/bin/arping' : File exists.

I think it is caused by the Makefile:

test -L "rootfs/$*/bin/$${COMMAND}" || ln -rs $< "rootfs/$*/bin/$${COMMAND}" 

The commands before 'arping' are all symbolic links, but 'arping' is not.
So it continues to run 'ln -rs' and returns an error.

We can change it to

test "rootfs/$*/bin/$${COMMAND}" || ln -rs $< "rootfs/$*/bin/$${COMMAND}" 

[alban]

@liangchenye did you forget the flag for the test command? test seems to always return true (0) without flags.

[liangchenye]

@alban thanks, so it should be

 test -e "rootfs/$*/bin/$${COMMAND}" || ln -rs $< "rootfs/$*/bin/$${COMMAND}"

[wking]

The commands before 'arping' are all symbolic links, but 'arping' is not.

Do you know where your arping came from? Everything in /bin should be a symlink to busybox except /bin/busybox itself.

[liangchenye]

I checked again, still failed.
I downloaded this file: http://distfiles.gentoo.org/releases/amd64/autobuilds/20180311T214502Z/stage3-amd64-20180311T214502Z.tar.xz which comes from latest-stage3.txt.
Its sha256sum is af849ce65244ee6dd1ef2a75deefe143933e82bce7d46bfcb24e36413cb5455e4f50f1d5cb887dc8cef84f70c2802ca1f09664b6d71cd3f129926d3dfa922424 stage3-amd64-20180311T214502Z.tar.xz.

Once I 'tar xvf' this file, there are only a few symlink files and most of them are not point to 'busybox'.
@wking

[wking]

On Tue, Mar 13, 2018 at 12:06:44PM +0000, 梁辰晔 (Liang Chenye) wrote: I downloaded this file: http://distfiles.gentoo.org/releases/amd64/autobuilds/20180311T214502Z/stage3-amd64-20180311T214502Z.tar.xz Its sha256sum is `af8…424 stage3-amd64-20180311T214502Z.tar.xz`. Once I 'tar xvf' this file, there are only a few symlink files and most of them are not point to 'busybox'.

That is the Gentoo stage3 pulled down by get-stage3.sh using this [1] Makefile target. The make rule should drop it into downloads/stage3-amd64-20180311T214502Z.tar.xz with a symlink downloads/stage3-amd64-current.tar pointing at downloads/stage3-amd64-20180311T214502Z.tar.xz. That's not the rootfs tarball though. The next step in building the rootfs tarball is to verify the signature and unpack anything matching the rootfs-files wildcards into rootfs/${ARCH}/bin/busybox [2]. Then we create symlinks for every command supported by that particular BusyBox binary [3]. And finally we tar it up into rootfs-${ARCH}.tar.gz [4]. So instead of looking at the Gentoo stage3 in downloads/, you should build and look at the final rootfs-${ARCH}.tar.gz. Steps to reproduce the tarballs I commit here are in the a7f94a2 commit message after “Generated with”. [1]: https://github.com/opencontainers/runtime-tools/blob/a7f94a268d2cbb5be522c433200aa44bd5f4d825/contrib/rootfs-builder/Makefile#L27-L29 [2]: https://github.com/opencontainers/runtime-tools/blob/a7f94a268d2cbb5be522c433200aa44bd5f4d825/contrib/rootfs-builder/Makefile#L8-L19 [3]: https://github.com/opencontainers/runtime-tools/blob/a7f94a268d2cbb5be522c433200aa44bd5f4d825/contrib/rootfs-builder/Makefile#L21-L25 [4]: https://github.com/opencontainers/runtime-tools/blob/a7f94a268d2cbb5be522c433200aa44bd5f4d825/contrib/rootfs-builder/Makefile#L5-L6

[liangchenye]

@wking yes, I do use the commands 'after “Generated with”'.

@alban @q384566678 how about you? can you reproduce it?

[alban]

The branch works for me. Here is the logs of my test:

$ git describe --tags
v0.5.0-48-ga7f94a2

$ make rootfs-amd64.tar.gz -C contrib/rootfs-builder/
make: Entering directory '/home/alban/go/src/github.com/opencontainers/runtime-tools/contrib/rootfs-builder'
STAGE3_ARCH=amd64 ./get-stage3.sh
--2018-03-14 11:49:47-- http://distfiles.gentoo.org/releases/amd64/autobuilds/latest-stage3.txt
Resolving distfiles.gentoo.org (distfiles.gentoo.org)... 140.211.166.134, 64.50.236.52, 137.226.34.46, ...
Connecting to distfiles.gentoo.org (distfiles.gentoo.org)|140.211.166.134|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1151 (1.1K) [text/plain]
Saving to: ‘STDOUT’
- 100%[==========================================================================================================>] 1.12K --.-KB/s in 0s 2018-03-14 11:49:47 (51.9 MB/s) - written to stdout [1151/1151]
--2018-03-14 11:49:47-- http://distfiles.gentoo.org/releases/amd64/autobuilds/20180313T214502Z/stage3-amd64-20180313T214502Z.tar.xz
Resolving distfiles.gentoo.org (distfiles.gentoo.org)... 140.211.166.134, 64.50.236.52, 137.226.34.46, ...
Connecting to distfiles.gentoo.org (distfiles.gentoo.org)|140.211.166.134|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 188872708 (180M) [application/x-xz]
Saving to: ‘downloads/stage3-amd64-20180313T214502Z.tar.xz’
downloads/stage3-amd64-20180313T214502Z.tar.xz 100%[==========================================================================================================>] 180.12M 1.01MB/s in 2m 56s 2018-03-14 11:52:43 (1.03 MB/s) - ‘downloads/stage3-amd64-20180313T214502Z.tar.xz’ saved [188872708/188872708]
--2018-03-14 11:52:43-- http://distfiles.gentoo.org/releases/amd64/autobuilds/20180313T214502Z/stage3-amd64-20180313T214502Z.tar.xz.CONTENTS
Resolving distfiles.gentoo.org (distfiles.gentoo.org)... 140.211.166.134, 64.50.236.52, 137.226.34.46, ...
Connecting to distfiles.gentoo.org (distfiles.gentoo.org)|140.211.166.134|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5241428 (5.0M) [application/x-xz]
Saving to: ‘downloads/stage3-amd64-20180313T214502Z.tar.xz.CONTENTS’
downloads/stage3-amd64-20180313T214502Z.tar.xz.C 100%[==========================================================================================================>] 5.00M 913KB/s in 8.6s 2018-03-14 11:52:52 (598 KB/s) - ‘downloads/stage3-amd64-20180313T214502Z.tar.xz.CONTENTS’ saved [5241428/5241428]
--2018-03-14 11:52:52-- http://distfiles.gentoo.org/releases/amd64/autobuilds/20180313T214502Z/stage3-amd64-20180313T214502Z.tar.xz.DIGESTS.asc
Resolving distfiles.gentoo.org (distfiles.gentoo.org)... 140.211.166.134, 64.50.236.52, 137.226.34.46, ...
Connecting to distfiles.gentoo.org (distfiles.gentoo.org)|140.211.166.134|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1630 (1.6K) [text/plain]
Saving to: ‘downloads/stage3-amd64-20180313T214502Z.tar.xz.DIGESTS.asc’
downloads/stage3-amd64-20180313T214502Z.tar.xz.D 100%[==========================================================================================================>] 1.59K --.-KB/s in 0s 2018-03-14 11:52:53 (50.3 MB/s) - ‘downloads/stage3-amd64-20180313T214502Z.tar.xz.DIGESTS.asc’ saved [1630/1630]
touch downloads/stage3-amd64-*.tar
gpg --verify downloads/stage3-amd64-current.tar.DIGESTS.asc
gpg: Signature made Wed 14 Mar 2018 02:24:09 CET using RSA key ID 2D182910
gpg: Good signature from "Gentoo Linux Release Engineering (Automated Weekly Release Key) <releng@gentoo.org>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 13EB BDBE DE7A 1277 5DFD B1BA BB57 2E0E 2D18 2910
(cd downloads && \
grep -A1 '^# SHA512 HASH' stage3-amd64-current.tar.DIGESTS.asc | \
grep -v '^--' | \
sha512sum -c)
stage3-amd64-20180313T214502Z.tar.xz: OK
stage3-amd64-20180313T214502Z.tar.xz.CONTENTS: OK
sudo rm -rf rootfs/amd64
sudo mkdir -p rootfs/amd64
sudo tar -xvf downloads/stage3-amd64-current.tar -C rootfs/amd64 \
--no-recursion --wildcards $(< rootfs-files)
./sys/
./bin/busybox
./dev/
./proc/
./etc/passwd
./etc/group
sudo touch rootfs/amd64/bin/busybox
sudo sh -c 'COMMANDS=$(rootfs/amd64/bin/busybox --list | grep -v "^busybox$") || exit 1; for COMMAND in ${COMMANDS}; do \
test -L "rootfs/amd64/bin/${COMMAND}" || ln -rs rootfs/amd64/bin/busybox "rootfs/amd64/bin/${COMMAND}" || exit; \
done'
tar -czf rootfs-amd64.tar.gz -C rootfs/amd64 .
rm downloads/stage3-amd64-current.tar
make: Leaving directory '/home/alban/go/src/github.com/opencontainers/runtime-tools/contrib/rootfs-builder'

$ sha512sum rootfs-amd64.tar.gz
d147a4a2da4a9257fc8cac44e56c05396f595c523cde6f588ebfeb6ddd5695b74ee8c3d59ac587727f35bf06d244abfc6ff0b89c175e37e2f796ec6d66511ce4 rootfs-amd64.tar.gz

$ tar tvf rootfs-amd64.tar.gz | grep '/arping\|/su$\|/mount$\|/passwd\|umount'
-rw-r--r-- root/root 655 2018-03-14 00:51 ./etc/passwd
lrwxrwxrwx root/root 0 2018-03-14 11:53 ./bin/arping -> busybox
lrwxrwxrwx root/root 0 2018-03-14 11:53 ./bin/umount -> busybox
lrwxrwxrwx root/root 0 2018-03-14 11:53 ./bin/passwd -> busybox

[liangchenye]

@alban I found that in my test, -wildcards $$(< rootfs-files) always return nil.
I guess it might be something wrong/difference in my Make/Makefile.

[liangchenye]

LGTM

[wking]

I guess it might be something wrong/difference in my Make/Makefile.

What version are you using?

[liangchenye]

GNU Make 3.81 on Ubuntu 14.04.