firmware: copy only the necessary firmware(s) into initramfs

[reagentoo]

FIRMWARE=yes behavior is changed:
Only the minimum number of firmware files will be copied.
The list is generated using the modinfo -F firmware [modules]... command.
The ability to copy all firmware(s) is also available with a new
ALLFIRMWARE setting (see the modified genkernel.conf for more details).

As for changes in the source code:
gen_moddeps.sh: Significantly redesigned module list generation.
To get a list of modules, use the mod_dep_list() function instead
of gen_dep_list(). Modules that are not in the kernel (=n or invalid)
will be filtered out. Aliases will be replaced with real names
(including dependencies).

[ZeroChaos-]

This is a great idea, but why hijack the original setting and make a new setting for the original way? How about a new MINFIRMWARE or something like that?

[reagentoo]

This is a great idea, but why hijack the original setting and make a new setting for the original way? How about a new MINFIRMWARE or something like that?

Because similar to RAMDISKMODULES / ALLRAMDISKMODULES.

[ZeroChaos-]

Changing (especially long term) behavior in such an incompatible way makes for a terrible user experience. I can't imagine that I am the only one who builds kernels and distributes them to multiple systems, but also there probably are not a lot of us either....

[reagentoo]

Compatibility preserved.

[reagentoo]

@FlyingWaffleDev, can you please check this patch on your configuration? In particular this:

PLYMOUTH="yes"
ALLRAMDISKMODULES="no"
RAMDISKMODULES="yes"
ALLFIRMWARE="no"
FIRMWARE="yes"
AMODULES_MISC="amdgpu i915"

(Or set your drm-module in AMODULES_MISC)
Need to make sure that the drm starts early and the plymouth works fine.
If you want to test and share results, then show also:

ls -1sh /boot/ | grep initramfs

In my case, the savings were ~300M for the last kernel (5.15.44), compared to what was mentioned in #29 (comment):

 12M initramfs-5.15.40-gentoo-dist.img
 12M initramfs-5.15.41-gentoo-dist.img
 14M initramfs-5.15.43-gentoo-dist.img
 27M initramfs-5.15.44-gentoo-dist.img
 27M initramfs-5.15.44-gentoo-dist.img.old

[FlyingWaffleDev]

@reagentoo Looks good! It works with my plymouth patch and saved me ~93M.

50M initramfs-5.17.12-gentoo-x86_64.img
143M initramfs-5.17.12-gentoo-x86_64.img.old

[robbat2]

does this work in the cross-compile kernels?

[reagentoo]

I tried to copy /lib/modules/*-gentoo-dist dir into /mnt/prefix, and modinfo -b ... -k ... worked.

[reagentoo]

I plan to make improvements to this place in the future. Something like:
modinfo ... || echo - handle bad status from modinfo in the while.
But for this I will need to make some changes to the gen_moddeps.sh. This will increase the predictability of the script. I think it would be better to make it a separate commit. Currently, everything should work reliably, unexpected errors will be logged and filtered in determine_initramfs_kmodules().

[robbat2]

I left a bunch of singleton comments.

I like the savings, but i'm also worried about the overall risk of unbootable systems.

There IS something useful that did stand out to me: can you double-check if the existing code captures dependency modules at all? e.g. modinfo -F depends amdgpu tells me drm_kms_helper,drm,gpu-sched,ttm,drm_ttm_helper,mfd-core,i2c-algo-bit.

[reagentoo]

There IS something useful that did stand out to me: can you double-check if the existing code captures dependency modules at all? e.g. modinfo -F depends amdgpu tells me drm_kms_helper,drm,gpu-sched,ttm,drm_ttm_helper,mfd-core,i2c-algo-bit.

This question about how gen_moddeps.sh works?
I did some analysis of this script. I wouldn't say it's perfect, but I think it captures all dependencies. At first I thought it was strange that there is no recursion to find all dependencies, but it looks like the modules.dep file contains an exhaustive number of dependencies for each line:

$ for i in $(cat /lib/modules/5.15.44-gentoo-dist/modules.dep | grep amdgpu); do basename $i | sed 's/.ko$//'; done | sort -u
amdgpu.ko:
cec
drm
drm_kms_helper
drm_ttm_helper
fb_sys_fops
gpu-sched
i2c-algo-bit
iommu_v2
syscopyarea
sysfillrect
sysimgblt
ttm

$ modinfo -F depends amdgpu | sed 's/,/\n/g' | sort -u
drm
drm_kms_helper
drm_ttm_helper
gpu-sched
i2c-algo-bit
iommu_v2
ttm

[reagentoo]

gen_moddeps.sh has been reworked (see new description). This had to be done as part of the current commit, because using the ALLRAMDISKMODULES=yes option did not guarantee correct copying of the firmware. I've tested to build ramdisk on my configuration. The number of modules copied and the ramdisk size have not changed.

[reagentoo]

@robbat2 Hi. What method of gen module deps do you prefer (modinfo -F depends or /lib/modules/${KV}/modules.dep) ?
I've suddenly noticed one feature in the module names. We have the following module, which is most often built as a non-builtin:

$ modinfo -F filename cciss
/lib/modules/5.15.48-gentoo-dist/kernel/drivers/scsi/hpsa.ko

$ modinfo -F filename hpsa
/lib/modules/5.15.48-gentoo-dist/kernel/drivers/scsi/hpsa.ko

$ cat /lib/modules/5.15.48-gentoo-dist/modules.dep | grep hpsa
kernel/drivers/scsi/hpsa.ko: kernel/drivers/scsi/scsi_transport_sas.ko

$ cat /lib/modules/5.15.48-gentoo-dist/modules.dep | grep cciss

$ modinfo -F depends cciss
scsi_transport_sas

$ modinfo -F depends hpsa
scsi_transport_sas

It looks like the same module has 2 names and one ko-file. (alias?)
If this is not an alias, then can there be a different list of firmware for such modules?

In the latest commit I've simplified the gen_moddeps.sh.
But with the current changes, cciss should be removed from the modules_load and left only hpsa. But I'm not sure.

UPD:
Looks like I answered myself. It's really an alias:

$ cat /lib/modules/5.15.48-gentoo-dist/modules.alias | grep cciss
alias cciss hpsa

[reagentoo]

@robbat2 Tested for several weeks. Looking forward for new comments on the new gen_moddeps.sh -
https://github.com/reagentoo/genkernel/blob/32f52927a7fce23958518ef7abdac7772f0aed0a/gen_moddeps.sh

[reagentoo]

@robbat2 Hi. Any chance to promote this commit?

[timkenhan]

Hello, I'd like to up this PR in hope to get more recognition.

Can we hopefully get this merged?

[reagentoo]

Hello, I'd like to up this PR in hope to get more recognition.

Can we hopefully get this merged?

@timkenhan Hi. Have you tried to apply and test it?
Can you share the results in the /boot dir (ls -1sh /boot/ | grep initramfs)?

[ZeroChaos-]

I don't work on genkernel much lately, but this all looks good to me.

[timkenhan]

@reagentoo here's the result:

 88M initramfs-6.1.46-gentoo-dist.img
 13M initramfs-6.1.46-gentoo-x86_64.img
267M initramfs-6.1.46-gentoo-x86_64.img.old

initramfs-6.1.46-gentoo-x86_64.img is built using genkernel-9999 with your changes. I also took the liberty to test out the Plymouth support as well. So far, there's no regression, and everything works as expected.

initramfs-6.1.46-gentoo-x86_64.img.old is built using the genkernel-9999 with no modification. Notice the size difference.

Last but not least, initramfs-6.1.46-gentoo-dist.img comes with gentoo-kernel package, built using dracut.

[reagentoo]

13M initramfs-6.1.46-gentoo-x86_64.img

@timkenhan Have you got HD picture in Plymouth? Maybe you don't have enough modules/firmwares.
My current initrd size:

 32M initramfs-6.1.47-gentoo-dist.img

/etc/genkernel.conf:

AMODULES_MISC="amdgpu i915"
COMPRESS_INITRD_TYPE="zstd"

[timkenhan]

@reagentoo

Not sure what HD picture should be like, but the boot splash does improve after applying your suggestion

After putting AMODULES_MISC="i915 nouveau amdgpu", it becomes:

 25M initramfs-6.1.46-gentoo-x86_64.img

Also can you document this AMODULES_MISC option? At least put it on the genkernel.conf. Might help other users as well.

Here's the list of files under initramfs lib/firmware with AMODULES_MISC and here's one without

[reagentoo]

Also can you document this AMODULES_MISC option? At least put it on the genkernel.conf. Might help other users as well.

This can be done in separate PR-s.

[thesamesam]

Thanks for your patience and thanks all for testing too.

Could you do the (unrelated) docs in a followup please?

[reagentoo]

Thanks for your patience and thanks all for testing too.

Thanks. Please assign me in bugzilla if we will have any related issues.

Could you do the (unrelated) docs in a followup please?

I'll prepare few commits for the new PR.