-
Notifications
You must be signed in to change notification settings - Fork 462
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Upgrade ubuntu image to 24.04 (LTS), alpine to 320 and ffmpeg to 7.1 #408
Upgrade ubuntu image to 24.04 (LTS), alpine to 320 and ffmpeg to 7.1 #408
Conversation
For reference I found/realized because of a SOC2 Vulnerabilities Scan on the docker image. |
converted to draft as I think this is going to require a Working through that now. |
alright I got
|
2404 is the current LTS, https://endoflife.date/ubuntu at this point if it's simpler to upgrade to Noble Numbat, I'd say let's drop 2204. Thank you very much for the contribution ! |
@jrottenberg concur, I will get this issue and pull-request cleaned up today to reflect the decision to add support for Ubuntu 2404 Noble Numbat, and I will leave all of the |
@jrottenberg you are welcome. Alright I cleaned up the PR and I think this is ready to go. |
Very nice, running it on https://dev.azure.com/video-tools/ffmpeg/_build/results?buildId=690&view=results |
Why 7.1 has its own env and run files ? I needed to change the download mirror for xvid, but that should be the main change required, rest should work for every version of ffmpeg, ideally. Trying to not have to do too much per version. |
|
I just chose the newest thing. ( just trying to make one thing work, for starters ) so: We can turn the knobs in To dial this back to get everything building. Looking Again
🤔 actually: this build (of the same branch): https://dev.azure.com/video-tools/ffmpeg/_build/results?buildId=691&view=results
based on this information: I am going to tweek the |
Doing some local builds to see if this works:
|
alright, after looking at the runs here Looks like everything built 🎉 except for the one Images that built
the Also: I went ahead and checked in 2 files in docker-images ( |
@jrottenberg If you re-kick the build, (picking up my latest 2 changes) it should build the Also pre commit on docker-images confusion
I cant do a fresh pull of my lates changes? ( that's weird)
This shows your latest commit, not the following 2 commits I made. In any case, maybe I should just check in the 167 Let me know what you think. Note: I made a PR review comment on |
Ah yeah, pre-commit is the first step of the pipeline so will fail and stop it if it hasn't run. Since I don't use docker for the builds, but azure pipeline, I wanted to make it very clear for users to lookup a given image and find the full Dockerfile that was used to generate it |
@jrottenberg I understand, this makes since. |
@jsheffie this PR does not yet resolve #403. The harfbuzz library and the I had planned a fix using the old method, but I’m waiting for this PR to be merged to use the new build system. You can test it with the command : |
FYI: This is because of an update to binutils. It's easy to fix via including this patch, I ran into this same problem before. |
This update breaks our CI workflow. I'm not 100% sure what the problem is, but it seems like copying /usr/local/lib and settings the LD_LIBRARY_PATH have something to do with it. Running the same workflow with a clean ubuntu 24.04 image and the repository FFmpeg version works fine. |
@Gwemox I added
Edited:
actually I think its has |
@jsheffie Is libfreetype also enabled? |
@Gwemox
here is what my local desktop build made. I think it should work. |
Hello @jsheffie, Indeed, drawtext seems to be active; however, there is an issue with the fonts.
|
Thank you very much @jsheffie ! Awesome work ! |
resolves: #407 , #406, #403
BLUF ( Bottom Line Up Front)
The following are ready to be built.
7.1
,7.02
,6.1.2
,and 5.1.6<ffmpeg-ver>-ubuntu24.04
<ffmpeg-ver>-ubuntu24.04-edge
<ffmpeg-ver>-vaapi24.04
<ffmpeg-ver>-nvidia24.04
<ffmpeg-ver>-alpine320
<ffmpeg-ver>-scratch320
Overview
I did not mean to have such a large commit. I did not. I actually originally had a one line fix... but turns out it was not that simple.
So here is how I got to where I am now. ( with such a large commit )
Simple enough, I started working through building the OS's with the lates libs. At some point along the way while working through the build / dependance issues, I realized ( after chasing my tail for awhile ) that I was getting intermittent download failures for some of the tarballs.
What was worse is that with the Docker
RUN DIR=/tmp/foo && \ mkdir ... && \ cd ... && \ curl ... && \ tar ... && \ configure && ...
construct it is very difficult to tell where the failure is. I do understand that we don't want the image to build in production if something fails. ( Note: it is sometimes useful to have the image finish building when in development if something fails so you can "peek under the hood" to figure out what went wrong. ( more on this later ) I decided there were several problem areas that needed to be addressed to keep this all easily maintainable. Another thought:Its a trade off, while building large portions form source allows us to run to the bleeding edge with the latest versions of all the libs, it comes at a difficult to maintain drawback. Most of these projects are being managed and regularly updated by debian and alpine. ( I do realize that this repo used to support other flavors of linux in the past )
Here is a screenshot from the README.md update that explains the image names.
ffmpeg-7.1-ubuntu2404
,ffmpeg-7.1-vaapi2404
,ffmpeg-7.1-alpine320
all use os vender libs for the ffmepg support libs. The advantage of this is that upgrading OS's should be a breeze in the future, as we call for generic vender lib install names. Examples are:libtheora-dev
,libtheora0
,libzmq3-dev
,libzmq5
... etc.We use the endoflife.date API to find the most recent ffmpeg versions. (which I thought was brilliant, by the way) However, to simplify image maintenance, we only consider versions released within the last YEARS years (currently set to 3). Including very old versions compatibility
issues with different libraries and operating system versions. By focusing on recent versions, we keep things manageable.
Note: the older builds will be preserved in the the docker hub registry.To keep this from getting even larger and more confusing I decided to support
The
Docerfile-env-<variant>
was a great abstraction, we need more of that. I realized when I got to the bottom of theREADME.md
with the package table to keep up to date, that we neededone place
to maintain this.Summary of changes:
download_tarballs.sh
script. Which will loop through the list and re-try things a configurable amount of times ( currently set to 6 ) If you feed the command several libs to download it will cycle through them before coming back and re-trying the one it failed on.build_sources.sh
bash script. Which adds aset -e
# Stop execution on any error
at the top of the bash script. Now when calling the build_sources.sh from a RUN line the build will fail ( the same as before )Docker-env-<variant>
. There were a few problems with this as well as a few details missing. It was repeated in the builds and the readme, and if we could store additional information about the packages we could use that information as well to, download, extract, checksum, and generate the table for theREADME.md
With those ideas in mind here is what a, Update the version of
libvmaf
tasking would look like../generate-source-of-truth-ffmpeg-versions.py
and modifyto the correct version and release date.
./generate-source-of-truth-ffmpeg-versions.py
generated_versions_table.md
into the bottom of theREADME.md
You probably should kick the tires on the image ( I updated the
CONTRIBUTING.md
with additional details on this )But here is a short walkthrough. I am currently developing on a Mac so the
--platform=
line is necessary,you might not need to do that.
Under the covers: We still have the
Docker-run-varient
In our example here you are looking forDocker-run-ubuntu-edge
You will notice the construct
# First batch of libraries ( split into docker layers, to allow for caching ) RUN /tmp/workdir/generate-source-of-truth-ffmpeg-versions.py --library-list \ libopencore-amr,\ libx264,\ libx265,\ libogg,\ libopus RUN /tmp/workdir/download_tarballs.sh RUN /tmp/workdir/build_source.sh
What this does is the first run line runs a python script inside of th development image which spits out
two manifests based on the lib arguments you give it
The next two RUN lines read in this information and do there operations.
download_tarballs.sh
will download ( w/ retries on failures ) the downloads.build_source.sh
has the source building callbacks, and will build ( in order ) the things you pass in.If any failures happen you the Docker image build will exit and the RUN line will fail straight away.
You can now string together these 3 RUN lines to process batches of libs to build in order. This will allow docker build to
cache the layers
of successful build items. ( this is pretty powerful when ironing out new any building / dependancies problems )Screenshot of Docker Caching build layers ( on re-builds ) 📸
Finally you can pass in
|| true
to any line to have itfail but keep going
you can use this trick to let the build finish so you can inspect the image.Rough Layout
Follow ups
docker-images
directory.docker-images
in the gitignore file. (but I did not do it in this commitThe files changed had ballooned to 167 files changed, so I went ahead and made this change. )Notes about initial changes
- add in support for ubuntu 2404 TLS - if using ffmpeg 7.1 or greater build with 2404. - added logic in update.py to build ffmpeg 7.1 and greater with new lib dependancies - added templates for `Dockerfile-env-ffmpeg-7.1-plus` and `Dockerfile-run-ffmpeg-7.1-plus` - these new templates updated these libs ( to make build work ) - OPENCOREAMR_VERSION `0.1.5` -> `0.1.6` - OPENJPEG_VERSION `2.1.2` -> `2.5.2` - X264_VERSION `20170226-2245-stable` -> `20191217-2245-stable` - X265_VERSION `3.4` -> `4.0` - XVID_VERSION `1.3.4` -> `1.3.7` - LIBZMQ_VERSION `4.3.2` -> `4.3.5` - LIBSRT_VERSION `1.4.1` -> `1.5.31 - LIBVMAF_VERSION `2.1.1` -> `2.3.1` - added template `Dockerfile-template.ubuntu2404` ( updated buildDeps to use `python3` ) - ~~removed template `Dockerfile-template.ubuntu2204` ( which was really pointed to `20.04` )~~ Decided not to remove 2204 at this time, so that this PR does not get too large.Developer notes
I was building this on my desktop w/ ( because I have apple silicon I needed the --platform argument )
Initial Description: was point Dockerfile-template.ubuntu2204 at correct version number
Docker template
Dockerfile-template.ubuntu2204
is pointed atFROM ubuntu:20.04 AS base
Which is old (March 2023), and misleading, since the template name specifies 2204.
should be
FROM ubuntu:22.04 AS base