add support to `eessi_container.sh` for using `unionfs` overlay tool by boegel · Pull Request #9 · EESSI/software-layer-scripts

boegel · 2025-06-14T13:53:28Z

Default behavior is left unchanged.

unionfs looks promising, haven't observed the Permission denied issues we've been seeing when creating overlay with fuse-overlays, but it needs to be battle-tested before we jump ship.

Tested with:

./eessi_container.sh --container docker://ghcr.io/eessi/build-node:debian12 --overlay-tool unionfs --access rw

(see also EESSI/filesystem-layer#226)

trz42

Minor suggestions

…iner.sh script Co-authored-by: Thomas Röblitz <trz42@users.noreply.github.com>

…r.sh

…layer-scripts into eessi_container_unionfs

boegel · 2025-06-17T10:00:21Z

- echo 'source /software-layer/scripts/utils.sh' >> ${test_script}
- echo 'source /software-layer/init/eessi_environment_variables' >> ${test_script}
+ echo 'ls -l /software-layer-scripts/' >> ${test_script}
+ echo 'sed -i "s/__EESSI_VERSION_DEFAULT__/${{matrix.EESSI_VERSION}}/g" init/eessi_defaults


actual fix is here, this looks more complicated than it should because of software-layer -> software-layer-scripts rename...

…AULT__ placeholder in init/eessi_defaults in CI workflow to test load_easybuild_module.sh script

bedroge · 2025-06-24T13:32:27Z

@boegel the CI is still failing because of:

ERROR: /cvmfs/software.eessi.io/versions/__EESSI_VERSION_DEFAULT__/compat/linux/x86_64 does not exist!

Any idea why the EESSI_VERSION_OVERRIDE is not working?

edit: quickly tried to reproduce that locally, but it worked fine for me...

boegel · 2025-06-24T16:41:28Z

@boegel the CI is still failing because of:
ERROR: /cvmfs/software.eessi.io/versions/__EESSI_VERSION_DEFAULT__/compat/linux/x86_64 does not exist!
Any idea why the EESSI_VERSION_OVERRIDE is not working?
edit: quickly tried to reproduce that locally, but it worked fine for me...

I've been trying to figure out why it's not working as it should, but I'm puzzled too...

bedroge · 2025-06-25T07:31:31Z

I tried it again, I think it's because run_in_compat_layer_env.sh does a source ${base_dir}/init/eessi_defaults. That's before you're overriding EESSI_VERSION_OVERRIDE, so then EESSI_VERSION gets set to that placeholder string.

Co-authored-by: Bob Dröge <b.e.droge@rug.nl>

… tests

…ipt tests

bedroge · 2025-06-25T12:03:53Z

 out="${PWD}/test_create_lmodsitepackage.out"
- ./eessi_container.sh --mode run --verbose /software-layer/run_in_compat_layer_env.sh /software-layer/test_lmod_sitepackage.sh 2>&1 | tee ${out}
- for pattern in "^Site Pkg location.*/software-layer/.lmod/SitePackage.lua" "LMOD_SITEPACKAGE_LOCATION.*/software-layer/.lmod/SitePackage.lua"; do
+ ./eessi_container.sh --mode run --verbose /software-layer-scripts/run_in_compat_layer_env.sh /software-layer-scripts/test_lmod_sitepackage.sh 2>&1 | tee ${out}


$EESSI_VERSION_OVERRIDE should also be set for this step

bedroge

Lgtm

add support to and for using unionfs overlay tool
9df2fbd

boegel added the enhancement New feature or request label Jun 14, 2025

trz42 requested changes Jun 16, 2025
View reviewed changes

Comment threadeessi_container.sh Outdated
Comment threadeessi_container.sh Outdated

boegeland others added 6 commits June 17, 2025 11:15

don't hardcode software.eessi.io when creating overlay in eessi_conta…
955d0e6
…iner.sh script Co-authored-by: Thomas Röblitz <trz42@users.noreply.github.com>

add test for using '--overlay-tool unionfs' option for eessi_containe…
1351105
…r.sh

add test for using '--overlay-tool unionfs' option for eessi_containe…
dd4cc2a
…r.sh

Merge branch 'main' into eessi_container_unionfs
e3a9eb8

replace EESSI version placeholder in CI workflow for scripts
b421f1d

Merge branch 'eessi_container_unionfs' of github.com:boegel/software-…
badfeba
…layer-scripts into eessi_container_unionfs

boegel commented Jun 17, 2025
View reviewed changes

boegel added 2 commits June 17, 2025 12:04

fix test of using unionfs as overlay-tool
d96fb6f

improve pattern check in test for using unionfs as overlay-tool
c9b81ba

boegel force-pushed the eessi_container_unionfs branch from 3aaa438 to c9b81baCompare June 17, 2025 18:53

boegel added 3 commits June 17, 2025 21:01

fix checking for use of '--overlay-tool unionfs'
2bbadca

fix replacing of EESSI version placeholder in CI workflow for scripts
8e2e292

set $EESSI_VERSION_OVERRIDE rather than replacing __EESSI_VERSION_DEF…
1675ff9
…AULT__ placeholder in init/eessi_defaults in CI workflow to test load_easybuild_module.sh script

boegel mentioned this pull request Jun 24, 2025
{2023.06}[2023a] Bazel/6.1.0 ml_dtypes/0.3.2 tensorboard/2.15.1 EESSI/software-layer#1131
Merged

boegel added 2 commits June 24, 2025 11:05

Merge branch 'main' into eessi_container_unionfs
1f11ae4

show contents of test script when testing load_easybuild_module.sh
00f3ea2

boegel mentioned this pull request Jun 24, 2025
Create utils function for nvidia-smi check #22
Merged

bedroge reviewed Jun 25, 2025
View reviewed changes

Comment thread.github/workflows/tests_scripts.yml

boegeland others added 3 commits June 25, 2025 10:15

set $EESSI_VERSION_OVERRIDE before calling run_in_compat_layer_env.sh
f9392e6
Co-authored-by: Bob Dröge <b.e.droge@rug.nl>

also set $EESSI_VERSION_OVERRIDE for install_software_layer.sh script…
dd37dfd
… tests

also set $EESSI_VERSION_OVERRIDE for create_directory_tarballs.sh scr…
5f4b557
…ipt tests

bedroge reviewed Jun 25, 2025
View reviewed changes

boegel mentioned this pull request Jun 27, 2025
update Java/11, Java/17, Java/21 wrapper modules to latest patch releases EESSI/software-layer#1102
Merged

bedroge reviewed Jun 27, 2025
View reviewed changes

Comment threadbot/build.sh Outdated

also set for create_lmodsitepackage.py tests
0ec75f1

boegel force-pushed the eessi_container_unionfs branch from a4e7200 to 0ec75f1Compare June 27, 2025 13:49

bedroge approved these changes Jun 27, 2025
View reviewed changes

bedroge merged commit 3ceef12 into EESSI:mainJun 27, 2025
47 checks passed

boegel deleted the eessi_container_unionfs branch June 27, 2025 14:13

This was referenced Jul 1, 2025
use unionfs overlay tool (+ debian12 build container that includes it) by default #26
Merged
Unable to rebuild on certain architectures EESSI/software-layer#1103
Closed

Navigation Menu

Uh oh!

add support to `eessi_container.sh` for using `unionfs` overlay tool - #9

add support to `eessi_container.sh` for using `unionfs` overlay tool#9
bedroge merged 18 commits into
EESSI:mainfrom
boegel:eessi_container_unionfs

boegel commented Jun 14, 2025

Uh oh!

trz42 left a comment

Uh oh!

Uh oh!

Uh oh!

boegelJun 17, 2025

Uh oh!

bedroge commented Jun 24, 2025•
edited
Loading

Uh oh!

boegel commented Jun 24, 2025

Uh oh!

bedroge commented Jun 25, 2025

Uh oh!

Uh oh!

bedrogeJun 25, 2025

Uh oh!

Uh oh!

bedroge left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

boegel commented Jun 14, 2025

Uh oh!

trz42 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

boegelJun 17, 2025

Choose a reason for hiding this comment

Uh oh!

bedroge commented Jun 24, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boegel commented Jun 24, 2025

Uh oh!

bedroge commented Jun 25, 2025

Uh oh!

Uh oh!

bedrogeJun 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bedroge left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bedroge commented Jun 24, 2025•
edited
Loading