image-creator

RaspberryPi image creator to build OLIP or Kiwix Hotspot off base-image.

Usage

image-creator is to be ran as root.

❯ image-creator --help
usage: image-creator [-h] [--build-dir BUILD_DIR] [--cache-dir CACHE_DIR] [-C] [-K] [-X] [-T CONCURRENCY] [-D] [-V] CONFIG_SRC OUTPUT
Create an Offspot Image from a config file
positional arguments:
CONFIG_SRC Offspot Config YAML file path or URL
OUTPUT Where to write image to
options:
-h, --help show this help message and exit
--build-dir BUILD_DIR
Directory to store temporary files in, like files that needs to be extracted. Defaults to some place within
/var/folders/p3/58pln35d7y15wpvvl49q3xpm0000gn/T
--cache-dir CACHE_DIR
Directory to use as a download cache. Should a remote file be present in the cache, it is fetched from there instead
of being downloaded. Files matching the cache policy are stored to the cache once downloaded. Cache Policy can be
configured in CACHE_DIR/policy.yaml
-C, --check Only check inputs, URLs and sizes. Don't download/create image.
-K, --keep [DEBUG] Don't remove output image if creation failed
-X, --overwrite Don't fail on existing output image: remove instead
-D, --debug
-V, --version show program's version number and exit
See https://github.com/offspot/image-creator for config and cache-policy format

Configuration

Image configuration is done through a YAML file which must match the following format. Only base is required.

Member	Kind	Function
base		Reference to the Offspot Base Image to use
base.source	string	Version (official releases) or URL to a base-image file. Accepts file:// URLs. Accepts lzma encoded images using .xz suffix
base.root_size	string/int	Size of the root (system) partition in the referenced base image (used to calculate free space)
output.size	string/int	Requested size of output image. Accepts auto for a cluster-aligned size that can fit the content
oci_images	image[]	List of OCI Image
image[].ident	string	specific OCI Image name. Prefer ghcr.io if possible. Format
image[].url	string	Optional URL to the exported tar file of the image. Downloaded from registry if not present
image[].filesize	int	Size in bytes of the exported tar file of the image
image[].fullsize	int	Size in bytes of the extracted tar file of the image. See Get OCI Image Sizes below
files	file[]	List of files to include on the data partition. See below. One of url or content must be present
files[].url	string	URL to download file from
files[].to	string	[required] Path to store file at. Must be a descendent of /data
files[].content	string	Text content of the file to write. Replaces url if present
files[].via	string	For url-based files, transformation to apply on downloaded file: direct (default): simple download, bztar, gztar, tar, xztar, zip to expand archives
files[].size	string/int	Only for *tar/zip should file be compressed. Specify expanded size. Assumes File-size (uncompressed) if not specified. ⚠️ Fails if lower than file size
files[].checksum	Checksum dict	Checksum definition for that file. See below.
write_config	bool	Whether to write this file to /data/conf/image.yaml
offspot	dict	runtime-config configuration. Will be parsed and dumped to /boot/offspot.yaml

Checksum

Member	Kind	Function
algo	string	Hashing algorithm. One of sha-1, sha-224, sha-384, sha-512, md5, adler32
value	string	Hex version of digest for algoritm or URL at which to read it (see kind)
kind	string	Either digest (default) or url, meaning value is an URL at which the value is readable as text

Note: When using a kind of url, the URL must return the content as text. Text must be only the digest or the digest followed by a space and arbitrary (ignored) text (to satisfy the md5sum –and alike– command output format, which includes the filename).

This URL feature is meant primarily to be used with MirrorBrain's automatic per-file-per-algorithm endpoints.

Sample

---
base: 1.0.0output:
size: 8Goci_images:
- ghcr.io/offspot/kiwix-serve:devfiles:
- url: http://download.kiwix.org/zim/wikipedia_fr_test.zimto: /data/contents/zims/wikipedia_fr_test.zimvia: direct
- to: /data/conf/message.txtcontent: | hello worldwite_config: trueoffspot:
timezone: Africa/Bamakoap:
ssid: Kiwix Offspotas-gateway: truedomain: demotld: offspotcontainers:
services:
kiwix:
container_name: kiwiximage: ghcr.io/offspot/kiwix-serve:devcommand: /bin/sh -c "kiwix-serve /data/*.zim"volumes:
- "/data/contents/zims:/data:ro"ports:
- "80:80"

Get OCI Images Sizes

image-creator needs to know about all content's sizes in order to create the image. This includes the OCI Image filesize and fullsize (once extracted on disk). To simplify getting those values, you can use a script that will download the image and extract it to compute the necessary sizes and print the required information in the appropriate format for copy-pasting.

curl -o ~/bin/get-oci-sizes https://raw.githubusercontent.com/offspot/image-creator/main/get-oci-sizes.py && chmod +x ~/bin/get-oci-sizes

Usage:

❯ ./get-oci-sizes.py ghcr.io/offspot/base-httpd:dev
Downloading ghcr.io/offspot/base-httpd:dev using docker-export
oci_images:
- id: ghcr.io/offspot/base-httpd:dev
# url: PREFIX/ghcr.io_offspot_base-httpd:dev.tar
filesize: 5980160 # 5.70MiB
fullsize: 5944584 # 5.67MiB
❯ ./get-oci-sizes.py caddy-2.6.1-alpine_armv6.tar
oci_images:
- id: caddy-2.6.1-alpine_armv6 # !fixup# url: PREFIX/caddy-2.6.1-alpine_armv6.tar
filesize: 43847680 # 41.82MiB
fullsize: 43803832 # 41.77MiB

Cache Policy

As image-creator mostly moves content from remote locations into an image file, it features a configurable Cache so that users creating multiple images have an option to download-once and reuse.

The cache is optional and set to a particular directory using --cache-dir option. The cache is flexible and configurable via a policy.yaml file inside of it. A default cache-policy is created if not present.

Only define the properties you want. Everything's optionnal. Sub-levels are bound by upper level. If you define a filter, a pattern is mandatory. First matching filter is applied.

Key	Kind	Default	Function
enabled	bool	true	Use to disable the cache completely.
max_size	size	10GiB	Overalll maximum size for cache. 0 disables
max_age	duration		Duration after which an entry should be evicted (from added-date)
max_num	int		Max number of items to keep in cache. 0 disables
check_after	duration		Duration during which an entry should not be checked for outdacy
keep_identified_versions	int		Number of identified version to keep per entry (see below)
eviction	string	lru	Main eviction Strategy. One of oldest, newest, largest, smallest, lru
oci_images	dict		OCI-Images specific cache configuration
oci_images.enabled	bool	true	If false, no OCI Image is cached.
oci_images.max_size	size		Size of the OCI-Images cache. Must fit witin main cache size
oci_images.max_age	duration		Duration after which an OCI Image should be evicted
oci_images.max_num	int		Max number of OCI-Images to keep in cache.
oci_images.check_after	duration		Duration during which an entry should not be checked for outdacy
oci_images.keep_identified_versions	int		Number of identified version to keep per image (see below)
oci_images.eviction	string	lru	OCI Images Eviction Strategy
oci_images.filters	list		Patterns to override config for. First matched is applied. Options applies to all matched
.filters[].pattern	string		Regexp to match OCI Image identifier. ex: /kiwix/
.filters[].max_size	size		Max total size of cache for entries of this pattern
.filters[].max_age	duration		Duration after which entries of this pattern should be evicted
.filters[].max_num	int		Max number of cache entries for this pattern
.filters[].check_after	duration		Duration during which an entry should not be checked for outdacy
.filters[].keep_identified_versions	int		Number of identified version to keep per matching image (see below)
.filters[].eviction	string	lru	Eviction strategy for cache entries of this pattern
.filters[].ignore	bool	false	Don't cache entries of this pattern
files	dict		Files (content and base image) specific cache configuration
files.enabled	bool	true	If false, no file/base is cached.
files.max_size	size		Size of the Files/base cache. Must fit witin main cache size
files.max_age	duration		Duration after which Files/base should be evicted
files.max_num	int		Max number of files/base to keep in cache.
files.check_after	duration		Duration during which an entry should not be checked for outdacy
files.keep_identified_versions	int		Number of identified version to keep per file (see below)
files.eviction	string	lru	Files Eviction Strategy
files.filters	list		Patterns to override config for. First matched is applied. Options applies to all matched
.filters[].pattern	string		Regexp to match Files URLs. ex: ^https?//download.kiwix.org/zim/
.filters[].max_size	size		Max total size of cache for entries of this pattern
.filters[].max_age	duration		Duration after which entries of this pattern should be evicted
.filters[].max_num	int		Max number of cache entries for this pattern
.filters[].check_after	duration		Duration during which an entry should not be checked for outdacy
.filters[].keep_identified_versions	int		Number of identified version to keep per matching file (see below)
.filters[].eviction	string		Eviction strategy for cache entries of this pattern
.filters[].ignore	bool	false	Don't cache entries of this pattern

• size type is a parse-able file size string (1G, 2.4GiB) or 0 string.
• duration type is a parse-able timespan string (30d4w1y) or 0 string.
• identified version means same-image with different tag for OCI Images (looks for :xxx in filename) or same ZIM file with different period for a ZIM (looks for _YYYY-MM.zim in filename)

Default Policy

---
enabled: truemax_size: 10GiBeviction: lruoci_images:
enabled: trueeviction: lrufiles:
enabled: trueeviction: lru

Sample Policy

---
enabled: truemax_size: 0oci_images:
eviction: oldestfiles:
enabled: false