Uh oh!
There was an error while loading. Please reload this page.
Allow a custom FFmpeg build to be provided using CMake variables - #1970
Conversation
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
ReenigneArcher
commented
Jan 1, 2024
This is a long shot, but maybe you could help with this? I'm trying to install boost from source on our macos build to save ~20 minutes of CI time. Somehow openssl headers aren't able to be found after this change. #1954 (comment) |
chewi
commented
Jan 1, 2024
Sorry, I deal with a number of environments but practically never encounter Macs. You had a hack to fix the OpenSSL headers before, perhaps that needs removing or adjusting now that you're explicitly installing it. |
2412aed to
0561448Compare| ${FFMPEG_PREPARED_BINARIES}/lib/libSvtAv1Enc.a | ||
| ${FFMPEG_PREPARED_BINARIES}/lib/libswscale.a | ||
| ${FFMPEG_PREPARED_BINARIES}/lib/libx264.a | ||
| ${FFMPEG_PREPARED_BINARIES}/lib/libx265.a | ||
| ${HDR10_PLUS_LIBRARY} |
There was a problem hiding this comment.
I'm a bit concerned about not including these libraries. What's the reason for removing them? Licensing?
There was a problem hiding this comment.
No, we're dynamically linking them via FFMPEG_PLATFORM_LIBRARIES, although this is also optional via Gentoo's USE flags. I have tested without them. If someone else wanted to statically link these, I suppose they could still provide them via FFMPEG_PLATFORM_LIBRARIES, using their full paths if necessary, but I haven't tried that.
Regarding HDR, I believe we (optionally) bake this support into the same libx265.so as the non-HDR support.
There was a problem hiding this comment.
Perhaps we should use a FindFFmpeg.cmake module instead?
- https://github.com/obsproject/obs-studio/blob/master/cmake/Modules/FindFFmpeg.cmake
- https://gitlab.kitware.com/vtk/vtk/-/blob/master/CMake/FindFFMPEG.cmake
- https://github.com/snikulov/cmake-modules/blob/master/FindFFmpeg.cmake
- https://sources.debian.org/src/acoustid-fingerprinter/0.6-6/cmake/modules/FindFFmpeg.cmake/
I think the one from OBS would be a preferred starting point (probably just works as is)
There was a problem hiding this comment.
I've only looked at the OBS one, but I don't think this would work. All these modules probably assume that the ffmpeg it finds is either a set of dynamic libraries or a set of entirely self-contained static libraries. They would not know that libx264 is also needed, for example. You'd also need to be careful not to pick up a vanilla system ffmpeg.
To be honest, I'm not a big CMake fan and I particularly don't like the find modules. pkg-config alone can handle this much better. In this case, it's a little awkward because you want this to work with your custom ffmpeg build out of the box, and a user could be building from any directory. To avoid needing to run something prior to CMake, you'd need to use configure_file to generate the right .pc file on the fly. Something like this:
Name: FFmpeg for Sunshine
Description:
Version: 0
Libs: -L@CMAKE_SOURCE_DIR@/third-party/ffmpeg/lib -lva -lva-drm -lx264 -x265
Cflags: -I@CMAKE_SOURCE_DIR@/third-party/ffmpeg/include
Gentoo could provide its own up front, and you could generate the above if an existing .pc file is not found. I'm not entirely sure this will work, as configure_file might not actually write the file early enough, but I'd be willing to give it a shot. Let me know what you think.
There was a problem hiding this comment.
I gave this a go, and it was looking quite good, but then fell apart when I tried cross-compiling. pkg-config can handle cross-compiling very well under Gentoo, but it gets upset if you try to use files outside of the cross environment like in this case.
If you still want to clean this up a bit, renaming your directories in build-deps could make it simpler:
set(FFMPEG_PREFIX ${CMAKE_SOURCE_DIR}/third-party/build-deps/ffmpeg/${CMAKE_SYSTEM_NAME}-${CMAKE_SYSTEM_PROCESSOR})
if(NOTIS_DIRECTORY${FFMPEG_PREFIX})
message(FATAL_ERROR"Unsupported operating system (${CMAKE_SYSTEM_NAME}) and processor (${CMAKE_SYSTEM_PROCESSOR}) combination")
endif()There was a problem hiding this comment.
when I tried cross-compiling
I'd love to have some guidance on this, or info added to the docs. It could possibly save us many hours of CI time for arm builds.
There was a problem hiding this comment.
I noticed that pkgconf (the modern pkg-config implementation) has a --define-prefix feature that might solve the issue I was having above. This might be good for Gentoo in general, so I'll look into that.
I believe the ${CMAKE_SYSTEM_NAME}-${CMAKE_SYSTEM_PROCESSOR} pairs would be:
- Windows-AMD64
- Darwin-x86_64
- Darwin-arm64
- Linux-x86_64
- Linux-aarch64
- Linux-ppc64le (you were also matching ppc64, but it's not ABI-compatible with your binaries)
I see you've been doing ARM builds with QEMU. That hurts. I've done very little cross-compiling outside of Gentoo, and I don't know much about GitHub Actions, but maybe I can offer some advice. I'll get back to you on that.
There was a problem hiding this comment.
I believe the ${CMAKE_SYSTEM_NAME}-${CMAKE_SYSTEM_PROCESSOR} pairs would be
Cool, I'll work on changing this over.
I see you've been doing ARM builds with QEMU.
It's actually not too bad for the Debian docker images. Those take ~45 minutes, which is somewhat bearable. But for some reason the Fedora images take 4-5 hours. For the life of me I cannot determine why they are so much slower.
I don't know much about GitHub Actions
It's really not much more than running shell/terminal commands. So if you know how to do it in Linux terminal, that's all I would need to try to incorporate it.
There was a problem hiding this comment.
You don't seem to have reorganised ffmpeg yet. Can we merge this in the meantime since it isn't dependent on that?
There was a problem hiding this comment.
Sorry for the delay, it's on my list of things to do.
e116946 to
d576f9eCompare
ReenigneArcher
left a comment
There was a problem hiding this comment.
I guess I'm approving this, with the disclaimer that using FFMPEG_PREPARED_BINARIES will never be checked/validated in our CI... so breakages may be likely from time to time.
Also, it's not ideal to use different versions of ffmpeg or the libraries that go along with it as it will increase the burden on support. If a user comes to us for an encoding issue and they are using Gentoo, where should we send them?
chewi
commented
May 13, 2024
Of course, I am perfectly happy to fix any breakage. Having this merged still reduces my own maintenance burden. Because the system-wide ffmpeg build cannot be used, the versioned Gentoo Sunshine package uses exactly the same ffmpeg release for everybody, currently 6.1.1. The "live" package, which is built from git and is understood to be potentially unstable, uses most of the submodules as you have them. Apart from being able to disable CUDA, AV1, VA-API, or x26[45], ffmpeg is also built the same way for everybody. I therefore don't anticipate any Gentoo-specific problems in this area, but the package does instruct users to contact us first, just as we agreed. If anyone comes here first, then please send them back to bugs.gentoo.org. |
It's only needed if libx265 was built with NUMA support. This support may be disabled in a custom FFmpeg build.
b2ce12c to
a9d3138CompareCodecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@## nightly #1970 +/- ##
==========================================
- Coverage 6.99% 6.82% -0.17%
==========================================
Files 87 87 Lines 17679 17678 -1 Branches 8399 8399 ==========================================
- Hits 1237 1207 -30 - Misses 13808 15781 +1973 + Partials 2634 690 -1944
Flags with carried forward coverage won't be shown. Click here to find out more. |
Description
For the Gentoo Linux package, we need to be able to use our own build of FFmpeg. We do not use the existing system installation as I understand that is sadly not possible. We do apply your patches for FFmpeg itself and libcbs.
I wanted to add CMake
options here for completeness, but then realised they're only for booleans. I considered globbing for*.afor more flexibility, but the linking order does matter. It's just a coincidence that the alphanumeric order happens to be the correct order at present.This is best viewed without whitespace changes.
Type of Change
.github/...)Checklist
Branch Updates
LizardByte requires that branches be up-to-date before merging. This means that after any PR is merged, this branch
must be updated before it can be merged. You must also
Allow edits from maintainers.