HDR screenshots are unimplemented (gpu/gpu-next)

[low-batt]

mpv Information

mpv v0.39.0-129-g4c24eae220 Copyright © 2000-2024 mpv/MPlayer/mplayer2 projects
 built on Oct 15 2024 20:49:12
libplacebo version: v7.349.0 (v7.349.0-18-g9e16c86f)
FFmpeg version: n7.1
FFmpeg library versions:
   libavcodec      61.19.100
   libavdevice     61.3.100
   libavfilter     10.4.100
   libavformat     61.7.100
   libavutil       59.39.100
   libswresample   5.3.100
   libswscale      8.3.100

Other Information

- macOS version: macOS 13.6.9 (22G830)
- Source of mpv: My own build of master using mpv-build
- Introduced in version: Unknown

Reproduction Steps

The version of IINA under development used to generate JPEG XL screenshots that used HDR color spaces. I recently tested it and found the screenshots looked muddy.

Running mpv on a Mac that supports EDR with --vo=libmpv and setting other options similar to what IINA does causes the video to be displayed with EDR enabled:

mpv/build/mpv --no-config --gpu-debug --log-file=output.txt --autofit=100% --cocoa-cb-output-csp=bt.2100-pq  --pause --screenshot-format=jxl --start=6 --target-prim=bt.2020 --target-trc=pq --vo=libmpv ~/Movies/The\ World\ in\ HDR\ in\ 4K\ \(ULTRA\ HD\).mp4

Take a screenshot.
Use jxlinfo to inspect the resulting JPEG XL screenshot.
Notice the screenshot uses sRGB.

Expected Behavior

A HDR screenshot would be generated.

Actual Behavior

The resulting JPEG XL screenshot:

low-batt@gag mpv-build (master %=)$ jxlinfo mpv-shot0001.jxl 
JPEG XL file format container (ISO/IEC 18181-2)
JPEG XL image, 3840x2160, lossy, 16-bit RGB+Alpha
Color space: RGB, D65, sRGB primaries, sRGB transfer function, rendering intent: Relative
low-batt@gag mpv-build (master %=)$ 

From log file:

[   6.457][v][cplayer] Starting screenshot: 'mpv-shot0001.jxl'
[   6.457][v][screenshot] input: 3840x2160 [0:0] rgba64 rgb/bt.709/srgb/full/display CL=mpeg1/jpeg
[   6.457][d][cplayer] Run command: script-binding, flags=73, args=[name="osc/__keybinding4"]
[   6.458][v][screenshot] will convert image to rgba64
[   6.458][v][screenshot] using 8 threads for scaling
[   6.461][v][screenshot] Using zimg.
[   6.475][d][screenshot] mapped color params:
[   6.475][d][screenshot]   trc = iec61966-2-1
[   6.475][d][screenshot]   primaries = bt709
[   6.475][d][screenshot]   range = pc
[   6.475][d][screenshot]   colorspace = gbr
[   6.475][d][screenshot]   chroma_location = center
[   7.137][i][cplayer] Screenshot: 'mpv-shot0001.jxl'

Adding --screenshot-sw=yes and repeating the test (looks better than this when viewed with a JPEG XL viewer):

low-batt@gag mpv-build (master %=)$ jxlinfo mpv-shot0002.jxl 
JPEG XL image, 3840x2160, lossy, 16-bit RGB
intensity_target: 10000.000000 nits
min_nits: 0.000000
relative_to_max_display: 0
linear_below: 0.000000
Color space: RGB, D65, Rec.2100 primaries, PQ transfer function, rendering intent: Relative
low-batt@gag mpv-build (master %=)$ 

For completeness, with gpu-next:

mpv/build/mpv --no-config --autofit=100% --pause --screenshot-format=jxl --start=6 --target-colorspace-hint --vo=gpu-next ~/Movies/The\ World\ in\ HDR\ in\ 4K\ \(ULTRA\ HD\).mp4

EDR is activated and the resulting screenshot is similar to libmpv with screenshot-sw, but slightly different:

low-batt@gag mpv-build (master %=)$ jxlinfo mpv-shot0003.jxl 
JPEG XL file format container (ISO/IEC 18181-2)
JPEG XL image, 3840x2160, lossy, 16-bit RGB+Alpha
intensity_target: 10000.000000 nits
min_nits: 0.000000
relative_to_max_display: 0
linear_below: 0.000000
Color space: RGB, D65, Rec.2100 primaries, PQ transfer function, rendering intent: Relative

Log File

output.txt
Screenshots.zip

Sample Files

I used a downloaded copy of this YouTube video: The World in HDR in 4K (ULTRA HD)

I carefully read all instruction and confirm that I did the following:

• I tested with the latest mpv version to validate that the issue is not already fixed.
• I provided all required information including system and mpv version.
• I produced the log file with the exact same set of files, parameters, and conditions used in "Reproduction Steps", with the addition of --log-file=output.txt.
• I produced the log file while the behaviors described in "Actual Behavior" were actively observed.
• I attached the full, untruncated log file.
• I attached the backtrace in the case of a crash.

[sfan5]

Is your intent to take a screenshot that is itself HDR or just a (tonemapped) screenshot of HDR video? I'm assuming the former.

I might be wrong but storing an HDR image as RGB sounds fundamentally wrong to me.
Since VO screenshots always return RGB that means this could possibly only work with --screenshot-sw --screenshot-tag-colorspace --screenshot-high-bit-depth and a format that takes YUV >=10-bits.
Note that ffmpeg's JXL support doesn't take YUV so is unsuited for this.
AVIF might work.

---

If you want to the latter:
In case of --screenshot-sw=no the VO probably returns either mistagged RGB or RGB but with a HDR csp. seems weird and broken??
You could argue that it's the VO's responsibility to produce a tonemapped screenshot even if the native display is HDR (how will this be configured?).
In case of --screenshot-sw=yes we'd need to tonemap in software with sws or zimg. Problem is neither can.
zimg produces the least bad but still incorrect result, probably what you were seeing with "Adding --screenshot-sw=yes and repeating the test".

[low-batt]

The intent is to obtain a HDR screenshot that matches what is seen on the screen, not a tone mapped screenshot. A JPEG XL screenshot that would activate EDR (with a viewer and a Mac that supports it). That is what is generated by --vo=gpu-next and what is generated with --vo=libmpv if I add --screenshot-sw=yes.

I believe at some point that is what mpv was generating without --screenshot-sw=yes as I don't remember needing to use that option when I was originally testing JPEG XL screenshot generation.

[haasn]

This is a vo_gpu limitation, no? It doesn't implement HDR screenshots. Shouldn't be too hard to support.

As a possible short term work around, we could fallback to sw screenshots if the VO does not support HDR screenshots.

[kasper93]

I have branch with refactored screenshot code in mpv to resolve some common issues. Though was too lazy to finish it. Maybe I will get back to it. In the meantime if someone wants to fix vo_gpu and others, don't hesitate.

[sfan5]

so to be clear the screenshot under

Adding --screenshot-sw=yes and repeating the test (looks better than this when viewed with a JPEG XL viewer):

is perfectly fine? and RGB with a HDR colorspace is no problem?

[haasn]

is perfectly fine? and RGB with a HDR colorspace is no problem?

JXL converts everything to XYZ anyway, doesn't it? So that's just one less conversion step.

[low-batt]

I played the movie with --vo=libmpv and --screenshot-sw=yes:

mpv/build/mpv --no-config --autofit=50% --cocoa-cb-output-csp=bt.2100-pq --script-opts-append=osc-visibility=always --pause --screenshot-format=jxl --screenshot-sw=yes --start=6 --target-prim=bt.2020 --target-trc=pq --vo=libmpv ~/Movies/The\ World\ in\ HDR\ in\ 4K\ \(ULTRA\ HD\).mp4

Took a screenshot and then displayed the screenshot:

mpv/build/mpv --no-config --autofit=50% --cocoa-cb-output-csp=bt.2100-pq --pause --script-opts-append=osc-visibility=always --target-prim=bt.2020 --target-trc=pq --vo=libmpv mpv-shot0001.jxl

I then took a picture of my MacBook Pro (XDR screen). The screenshot is on top. To my eyes the screenshot looks like the video:
IMG_1070.HEIC.zip