Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to FFMPEG 5.1.2 and fix heroku-22 execution #31

Merged
merged 10 commits into from
Mar 8, 2023
Merged

Conversation

schneems
Copy link
Contributor

@schneems schneems commented Feb 22, 2023

Builds off of the work of #28 to download a binary from S3 (versus previously we built a static binary as a .deb. This should close #27 when merged, and also close #24.

In addition caching is removed as both the cache and binary live on S3. Removing caching will decrease cache size and not affect build performance.

This PR is a continuation from the original discussion at #30. The main difference is the branch was renamed for easier use via heroku buildpacks

Warning

This change is gated on the full rollout of https://devcenter.heroku.com/changelog-items/2547 as #28 relies on updated system packages to be present. The common runtime and private spaces runtime update their stack images at different intervals. Due to this, merge will be delayed.

Herokai: Do not approve until dogwood has confirmed stack image rollout.

Use this branch without a merge

Developers can test use this branch before merge by running:

heroku buildpacks:remove https://github.com/heroku/heroku-buildpack-activestorage-preview
heroku buildpacks:add -i 1 https://github.com/heroku/heroku-buildpack-activestorage-preview#update-ffmpeg

Once merged, you should switch back or your app will not receive updates or security fixes. You can switch back by running:

heroku buildpacks:remove https://github.com/heroku/heroku-buildpack-activestorage-preview#update-ffmpeg
heroku buildpacks:add -i 1 https://github.com/heroku/heroku-buildpack-activestorage-preview

If you're using pipelines, make sure you run commands on your staging app so binaries are properly promoted.

GUS-W-12618647

@dzuelke
Copy link
Contributor

dzuelke commented Feb 24, 2023

Merged #28; so this needs a rebase. Also added a CHANGELOG.md entry for that PR, so you can remove one of your two changelog lines, @schneems.

@schneems
Copy link
Contributor Author

Updated

@jrochkind
Copy link

jrochkind commented Feb 27, 2023

Hi!

I have confirmed that using buildpack https://github.com/heroku/heroku-buildpack-activestorage-preview#update-ffmpeg on heroku-22, I can now execute the ffprobe and ffmpeg commands that previously segfaulted for me.

A sample repro command: ffprobe https://scih-data-dev.s3.amazonaws.com/test_video/SampleVideo_360x240_1mb.mp4

I probably won't actually switch to heroku-22 until I can do this without using a non-standard branch of the activestorage-preview buildpack. I am not in a hurry, but curious if you can share any info on timeline.

If I remove the activestorage-preview buildpack entirely, and try to use the alternate third-party buildpack https://github.com/jonathanong/heroku-buildpack-ffmpeg-latest.git -- it previously worked on heroku-20, but segfaulted on heroku-22. Currently... this is still the case. That third-party buildpack works on heroku-20 but causes a segfault on heroku-22, as activestorage-preview does before this fix. While I understand you do not support third-party buildpacks, I wonder if this indicates a problem with heroku-22 that should be reported independently from the activestorage-preview buildpack?

Thank you!

@schneems
Copy link
Contributor Author

@jrochkind awesome! Thanks for the great feedback, and sorry to leave you waiting so long for an initial response.

@dzuelke
Copy link
Contributor

dzuelke commented Mar 1, 2023

If I remove the activestorage-preview buildpack entirely, and try to use the alternate third-party buildpack https://github.com/jonathanong/heroku-buildpack-ffmpeg-latest.git -- it previously worked on heroku-20, but segfaulted on heroku-22. Currently... this is still the case. That third-party buildpack works on heroku-20 but causes a segfault on heroku-22, as activestorage-preview does before this fix. While I understand you do not support third-party buildpacks, I wonder if this indicates a problem with heroku-22 that should be reported independently from the activestorage-preview buildpack?

This is not specific to Heroku-22; there are several reports on several of these third-party builds of FFMPEG (which the buildpack you mentioned uses) of segfaults with Ubuntu 22.04 in other environments, e.g. GitHub Actions.

The cause is the static linking of glibc - this can break DNS lookups via NSS, which is still used via dlopen() internally unless --enable-static-nss is used, but all of this static linking comes with a ton of pitfalls, which is why the new builds in this PR use dynamic linking.

We'll merge and release this PR this week, FYI.

@dzuelke dzuelke force-pushed the update-ffmpeg branch 2 times, most recently from 3b28cb3 to c9d4eaa Compare March 1, 2023 09:50
@dzuelke
Copy link
Contributor

dzuelke commented Mar 1, 2023

Rebased on top of main.

Once ready, please merge this using a regular merge commit to ease merging/rebasing of stacked PRs #32 and #34

This was referenced Mar 1, 2023
@jrochkind
Copy link

Thank you for the attention, and the info on ffmpeg and ubuntu 22 generally. I appreciate it!

Previously we were hosting a `.deb` with the updates from #28 we are now using a `tar.gz`.

We can get rid of the cache as we have to download from S3 either way. This change is gated on the full rollout of https://devcenter.heroku.com/changelog-items/2547 as #28 relies on updated system packages to be present.
- Remove cache of deb packages

Previously we stored a cache of the static ffmpeg binary, however both the cache and the binary are stored on S3 so this does not improve compile times. By moving to not use the cache and deleting old contents we can reduce cache size while preserving compile times

- Move location of `dpkg -x` to be logically consistent with installing poppler

Previously we were installing ffmpeg and poppler via the apt-get mechanism. With the move to have FFMPEG be a simple curl to disk, it makes more sense to have the `dpkg -x` logic be with the poppler download code.

- Add additional logging

Log download and install for poppler and ffmpegcache. Adds the version of ffmpeg to the log output.
@dzuelke dzuelke merged commit de98e19 into main Mar 8, 2023
@dzuelke dzuelke deleted the update-ffmpeg branch March 8, 2023 11:10
@andreas-venturini
Copy link

@schneems thanks, very helpful 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ffmpeg/ffprobe on heroku-22 will segfault with URL arguments ffmpeg version update?
5 participants