qt: preparation for qtmultimedia

[timblechmann]

This is a proof of concept for integrating qtmultimedia's fuzz tests into oss-fuzz.

it includes some cleanups:

• using init-repository to clone qt's submodules with their dependencies instead of cloning manually. note: this requires a --depth argument for shallow clones, which is not yet merged into qt5.git: https://codereview.qt-project.org/c/qt/qt5/+/760920 (it can probably not merged before landing this)
• moving to ubuntu 24.04 as 20.04 is not officially supported anymore and qtmultimedia will want newer ffmpeg. (it will require a few compile fixes in qt for newer libc++, which will require the submodule update bot to cycle)

the PR contains multiple independent patches. some could be landed independently, but it's probably best to land them all at once.

CC @rlohning

[github-actions]

timblechmann is a new contributor to projects/qt. The PR must be approved by known contributors before it can be merged. The past contributors are: rlohning, hunsche, DonggeLiu, sgaist, cvediver

[rlohning]

Hello Tim,
thank you for your contribution. I will look into it soon, but today I won't be able. Please don't merge anything without my approval.
Cheers,
Robert

[rlohning]

Please don't use init-repository. It often clones code which is several weeks old which is pretty unfortunate for continuous fuzzing.

I understand that building qtmultimedia would require qtdeclarative to be built, too. When I tried that a couple of years ago, I didn't succeed. I don't know what effort would be needed to do it now.

[rlohning]

I removed the use of init-repository in #7017.

[rlohning]

Thank you for spotting the issue with the old Ubuntu.

According to the documentation it only takes this one line for porting to 24.04. If you add that as a separate pull request with this, I'll support it.

[timblechmann]

i'm not sure if it is sufficient, but also the docker file needs to be updated. at least that's what other projects are doing. e.g. 959e29f

[rlohning]

Good catch!

I'm asking that paragraph's author how to do it right.

[rlohning]

Both changes are needed, indeed.

[rlohning]

I split off the Ubuntu update to #16038

[timblechmann]

Please don't use init-repository. It often clones code which is several weeks old which is pretty unfortunate for continuous fuzzing.

I understand that building qtmultimedia would require qtdeclarative to be built, too. When I tried that a couple of years ago, I didn't succeed. I don't know what effort would be needed to do it now.

qtmultimedia will pull in a handful of dependencies, not only declarative. those will have dependencies on their own, so we will have to maintain the dependency lists ourselves.
that said, we could use init-repository --branch, no?

that said, there is no guarantee that the dev branches of the modules build against each other. this will hit qtmultimedia every once in a while due to the interaction of the rendering stack between qtbase, qtshadertools and qtmultimedia. extrapolating from the last few years, this is going to happen a few times per year and in the worst case we may end up with not being able to build for a few weeks (the price to pay for qt's multi-repo setup)

[rlohning]

Except for the Ubuntu update, which I extracted to #16038, I'm not convinced of this, yet. I see several issues which we should discuss separately:

• You use the sources init-repository clones, i.e. the submodule's revisions which are listed in qt5.git
  These get updated only after all submodules passed a dependency update. That means the sources may be several weeks old. I don't see a point in waiting for dependency updates to modules which we don't use here anyway, e.g. qtvirtualkeyboard or qt3d.

• You rightfully point out that checking out all submodules' dev branches might not compile.
  That's a solved problem. We have been using cmake/QtSynchronizeRepo.cmake for years now. We'd only have to use -DSYNC_TO_MODULE=qtmultimedia instead of qtsvg. That will still give us older sources but they may be newer than what is in qt5.git.

• You use seem to prefer the convenience of init-repository cloning all dependencies compared to cloning the submodules manually.
  I admit you have a point there. Nevertheless I'm not convinced that this convenience is worth the change. oss-fuzz lists revision ranges for each git repository when regressions or fixes happened. I don't want to risk losing this information by just having one repository with submodules.
  Cloning the modules ourself is a one time effort. We'd only have to add clones for all of qtmultimedia's dependencies. I don't expect those to change that often.

[timblechmann]

You use the sources init-repository clones, i.e. the submodule's revisions which are listed in qt5.git

--branch would check out the dev branch

That's a solved problem. We have been using cmake/QtSynchronizeRepo.cmake for years now. We'd only have to use -DSYNC_TO_MODULE=qtmultimedia instead of qtsvg.

i'm curious: how is this solved? if we have component A, B, C and D. C requires revision B-v1, D requires revision B-v2. B, C and D will provide fuzzing targets. they may not be compatible. if we want to fuzz B, C and D, we would be better off splitting all of them into separate fuzzing projects and building them independently.
i'm honestly seeing this as a real world problem here: qtdeclarative and qtsvg have incompatible changes and need to be kept in sync. similar to the QRhi stack requiring qtbase, qtshadertools, qtdeclarative and qtmultimedia to be synced ... it does happen from time to time that changes in all these repos need to be aligned or there are will be build failures.

[rlohning]

--branch would check out the dev branch

It would check out each submodule's respective dev branch, it seems. You get an untested combination then.

i'm curious: how is this solved?

I don't know the internals. What I expect and what QtSynchronizeRepo.cmake seems to do is that it parses qtmultimedia's dependencies.yaml and checks out the revisions listed there. Then it recursively does the same in the modules it just checked out. Because each module's respective dependencies.yaml was updated after a passed CI run, this gives you a combination of revisions which passed CI before.

If you 'd like to know in more detail, Alexandru, Joerg or Volker can probably tell you.

[timblechmann]

What I expect and what QtSynchronizeRepo.cmake seems to do is that it parses qtmultimedia's dependencies.yaml and checks out the revisions listed there.

but qtsvg and qtimageformats are both dependencies of qtmultimedia (via qtdeclarative). so they will be synced to the dependencies.yaml of qtmultimedia then.

it sounds to me that the approach of fuzzing all of qt in the same checkout doesn't scale if we want to have commit granularity of the different submodules. also the current state will break if there are some source incompatibilities between qtbase that qtsvg and qtimageformat requires ...

[timblechmann]

apparently we will need to use a separate project in oss-fuzz for qtmultimedia

[rlohning]

but qtsvg and qtimageformats are both dependencies of qtmultimedia (via qtdeclarative). so they will be synced to the dependencies.yaml of qtmultimedia then.

Yes. Isn't that exactly what you want?

qtrob@adlershoffice:/tmp/qt$ cmake -DSYNC_TO_MODULE=qtmultimedia -DSYNC_TO_BRANCH=dev -P cmake/QtSynchronizeRepo.cmake
Checking 'qtmultimedia' out to revision 'dev'
Checking 'qtquick3d' out to revision '4ba9cad410638a9780f004bd6f23b8c386041f6a'
Checking 'qtdeclarative' out to revision '88e2960cf1d403181adf852cf2ab4e61aefe1241'
Checking 'qtsvg' out to revision 'dc2c61b341dc364e54fc93768a96d505111a6428'
Checking 'qtshadertools' out to revision '2a6170db853a7f39f6239d6828b33466a2c73d55'
Checking 'qtlanguageserver' out to revision '179ae2cb8ad5dedc5e74f49ec709a3cd50067e1a'
Checking 'qtimageformats' out to revision '75f20b985e23bdf22c9d46e4f390e378be19d2ca'
Checking 'qtbase' out to revision 'b0fea7016954821251a7e493c7c3f47a5f013f26'

We'd be testing a somewhat older qtsvg than we do with the current script but it's a consistent, tested combination of revisions.