[Bug]: ExitCode 137/139 when uploading .mov videos

[don-vip]

Contact Details

No response

What happened?

I cannot upload any .mov video from https://svs.gsfc.nasa.gov/20411/

All tasks fail with exit code 137/139.

What browser(s) are you experiencing the problem on?

No response

Relevant error messages

A Pathway to Protocells on Titan – Animations (SVS20411 - TitanProtoCell Lake Descent)	An exception occurred: TaskError: /mnt/nfs/labstore-secondary-project/gentoo-prefix/usr/bin/ffmpeg -y -i /srv/v2c/output/11004e056113ab30/dl.mov -max_muxing_queue_size 4096 -movflags use_metadata_tags -map_metadata 0 -threads 8 -crf 30 -b:v 0 -vcodec libsvtav1 -svtav1-params lp=8 -preset 6 -f webm -ss 0 -an /srv/v2c/output/11004e056113ab30/dl.mov.an.av1.webm Exitcode: 137

A Pathway to Protocells on Titan – Animations (SVS20411 - TitanProtoCell Rain)	An exception occurred: TaskError: /mnt/nfs/labstore-secondary-project/gentoo-prefix/usr/bin/ffmpeg -y -i /srv/v2c/output/5e730fabb577d834/dl.mov -max_muxing_queue_size 4096 -movflags use_metadata_tags -map_metadata 0 -threads 8 -crf 30 -b:v 0 -vcodec libsvtav1 -svtav1-params lp=8 -preset 6 -f webm -ss 0 -an /srv/v2c/output/5e730fabb577d834/dl.mov.an.av1.webm Exitcode: 139

A Pathway to Protocells on Titan – Animations (SVS20411 - TitanProtoCell Acrylonitrile Shot)	An exception occurred: TaskError: /mnt/nfs/labstore-secondary-project/gentoo-prefix/usr/bin/ffmpeg -y -i /srv/v2c/output/29917aeba2b606ce/dl.mov -max_muxing_queue_size 4096 -movflags use_metadata_tags -map_metadata 0 -threads 8 -crf 30 -b:v 0 -vcodec libsvtav1 -svtav1-params lp=8 -preset 6 -f webm -ss 0 -an /srv/v2c/output/29917aeba2b606ce/dl.mov.an.av1.webm Exitcode: 139

A Pathway to Protocells on Titan – Animations (SVS20411 - TitanProtoCell Molecule Shot)	An exception occurred: TaskError: /mnt/nfs/labstore-secondary-project/gentoo-prefix/usr/bin/ffmpeg -y -i /srv/v2c/output/f108316be7d6df9c/dl.mov -max_muxing_queue_size 4096 -movflags use_metadata_tags -map_metadata 0 -threads 8 -crf 30 -b:v 0 -vcodec libsvtav1 -svtav1-params lp=8 -preset 6 -f webm -ss 0 -an /srv/v2c/output/f108316be7d6df9c/dl.mov.an.av1.webm Exitcode: 139

A Pathway to Protocells on Titan – Animations (SVS20411 - TitanProtoCell Space Shot)	An exception occurred: TaskError: /mnt/nfs/labstore-secondary-project/gentoo-prefix/usr/bin/ffmpeg -y -i /srv/v2c/output/122246f52d7465c6/dl.mov -max_muxing_queue_size 4096 -movflags use_metadata_tags -map_metadata 0 -threads 8 -crf 30 -b:v 0 -vcodec libsvtav1 -svtav1-params lp=8 -preset 6 -f webm -ss 0 -an /srv/v2c/output/122246f52d7465c6/dl.mov.an.av1.webm Exitcode: 137

A Pathway to Protocells on Titan – Animations (SVS20411 - TitanProtoCell Droplet Shot pt3)	An exception occurred: WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM) Job: 13.

A Pathway to Protocells on Titan – Animations (SVS20411 - TitanProtoCell Droplet Shot pt1)	An exception occurred: TaskError: /mnt/nfs/labstore-secondary-project/gentoo-prefix/usr/bin/ffmpeg -y -i /srv/v2c/output/78476fd3020ee641/dl.mov -max_muxing_queue_size 4096 -movflags use_metadata_tags -map_metadata 0 -threads 8 -crf 30 -b:v 0 -vcodec libsvtav1 -svtav1-params lp=8 -preset 6 -f webm -b:a 128000 -ar 48000 -acodec libopus /srv/v2c/output/78476fd3020ee641/dl.mov.av1.webm Exitcode: 137

A Pathway to Protocells on Titan – Animations (SVS20411 - Cell Membrane Zoom In V3)	An exception occurred: TaskError: /mnt/nfs/labstore-secondary-project/gentoo-prefix/usr/bin/ffmpeg -y -i /srv/v2c/output/8ad31d97956bb2cc/dl.mov -max_muxing_queue_size 4096 -movflags use_metadata_tags -map_metadata 0 -threads 8 -crf 30 -b:v 0 -vcodec libsvtav1 -svtav1-params lp=8 -preset 6 -f webm -an /srv/v2c/output/8ad31d97956bb2cc/dl.mov.an.av1.webm Exitcode: 137

A Pathway to Protocells on Titan – Animations (SVS20411 - TitanProtoCell Droplet Shot pt2)	An exception occurred: TaskError: /mnt/nfs/labstore-secondary-project/gentoo-prefix/usr/bin/ffmpeg -y -i /srv/v2c/output/ab4d3b0a4ef88323/dl.mov -max_muxing_queue_size 4096 -movflags use_metadata_tags -map_metadata 0 -threads 8 -crf 30 -b:v 0 -vcodec libsvtav1 -svtav1-params lp=8 -preset 6 -f webm -an /srv/v2c/output/ab4d3b0a4ef88323/dl.mov.an.av1.webm Exitcode: 139

[Amdrel]

I tested one of these files and the memory usage from the encoding process on my machine plateaued at 10GB. This means that our 16GB workers can only handle one of these heavy files at a time.

I noticed these videos are very high resolution and use ProRes, both of which are likely contributing to the high memory usage. One way we can work around this problem is to detect which files are likely to use a substantial amount of memory by probing the resolution and/or the encoding, and route those videos so only workers that are configured to process a single task at a time can process it. I'm pretty sure Celery supports this with the addition of another queue. You could call the secondary queue 'heavy' and have all workers listening on that queue restrict their concurrency to 1.

The downside is it might take longer for all those videos to process, but it should hopefully fix the OOM errors that they're triggering. I'm going to explore this and see if I can get it working on my own setup.

[don-vip]

Sounds nice, I agree with your proposal 👍