Update big picture document and add sub-documents

canvas-compositing-headroom.md

@@ -0,0 +1,91 @@
+# 2D Canvas drawing HDR headroom parameter
+
+## Introduction
+
+This proposal suggest a mechanism for drawing HDR content as HDR to a `CanvasRenderingContext2D` or `OffscreenCanvasRenderingContext2D`.
+
+This fits into the [HDR on the web, the big picture](hdr-big-picture.md) proposal. It is the "red star number 1" problem in the proposal diagram.
+
+## Motivation
+
+### Remove ambiguity
+
+When drawing an image to an HDR canvas, using `drawImage`, the source image is converted from its native color space to the color space of the canvas.
+There is currently no indication about what tone mapping, if any, should be done during this conversion.
+
+### Enable drawing HDR images and video as HDR
+
+By default, all implementations tone map to SDR in the process of converting to the color space of the canvas.
+It should be possible to preserve HDR-ness of HDR images in the canvas, especially now that canvas' backing bitmaps can be floating-point.
+
+### Enable drawing HDR CSS colors as HDR
+
+HDR CSS colors (using [`color-hdr`](https://drafts.csswg.org/css-color-hdr/#hdr-color-function)) are parameterized by a target HDR headroom.
+When such colors are used with 2D canvas, this displaying their non-HDR representation.
+
+### Standardize tone mapping for SDR renditions of HDR images
+
+This behavior can be used to ensure that all browsers tone map HDR images to SDR in the same way.
+
+### Ubiquitous HDR images
+
+There is an enormous and growing corpus of ISO 21496-1 HDR gain map images being produced, as this is now the default format for many popular phones.
+
+## Related non-web APIs
+
+### Chromium internals
+
+The Chromium browsers have a piece of rendering state called the [target HDR headroom](https://source.chromium.org/chromium/chromium/src/+/main:cc/paint/paint_op_buffer.h;l=169;drc=87c3217dc3fec0f441b68f33d339b7f3a707b11d) that is set during rendering.
+When drawing an HDR image using `drawImage`, that parameter is used for tone mapping by Skia.
+
+### CoreGraphics API
+
+In CoreGraphics, there exists a similar API called [`CGContextSetEDRTargetHeadroom`](https://developer.apple.com/documentation/coregraphics/cgcontext/setedrtargetheadroom(_:)?language=objc), which is documented (in the header file) as "Set target EDR headroom on a context to be used when rendering HDR content to the context.".
+
+These two implementations are extremely similar, down to the variable names.
+
+## Separate work
+
+### Tone mapping at display time
+
+The HDR headroom used when rendering is completely independent from the tone mapping behavior.
+In practice, most applications will want to tone map taking into account the HDR headroom that was used for displaying the canvas.
+
+A [separate set of proposals](hdr-big-picture.md#displaying-an-hdr-canvas) address that issue.
+
+## Proposal
+
+Add to the mixin interface, `CanvasCompositing` the following attribute
+
+```idl
+  partial interface mixin CanvasCompositing {
+    // The linear HDR headroom to tone map content to when
+    // drawing to the canvas.
+    attribute unrestricted double globalLinearHDRHeadroom; // (default 1)
+  }
+```
+
+This attribute may be set to `Infinity` to indicate that the "maximum headroom" version is desired.
+
+Setting this attribute to value outside of the interval [`1`, `Infinity`] will have no effect.
+
+### Interactions with drawing
+
+The attribute will affect the rendering of images when drawn with `drawImage`.
+
+It will also affect the tone mapping of CSS colors specified by `color-hdr`.
+
+This will not affect how CSS colors specified without `color-hdr` are drawn to the canvas.
+E.g, the color `color(srgb 2 2 2)` will always write the value [2,2,2] to a canvas with `colorSpace` `srgb` and `colorType` `float16`.
+
+## Testing
+
+The initial tests for this image type should be HDR images that have tone mapping that is parameterized by HDR headroom.
+
+The most clearly specified version of this is ISO 21496-1 gain map images.
+A set of WPT tests that draw gain map images at multiple HDR headrooms should be added.
+
+A less clearly specified version of this is ISO 22028-5 images.
+The tone mapping at HDR headroom of 0 (SDR) is defined in ISO 22208-5, but tone mapping to other headrooms is not specified.
+A set of WPT tests that draw these images at HDR headroom 0 (SDR) and their maximum HDR headroom should be added.
+

canvas-smpte-st-2094-50.md

@@ -0,0 +1,69 @@
+# SMPTE ST 2094-50 Tone mapping for canvas elements
+
+This proposal suggests an interface for adding support for [SMPTE ST 2094-50](https://github.com/SMPTE/st2094-50) tone mapping to `CanvasToneMapping`.
+
+This fits into the [HDR on the web, the big picture](hdr-big-picture.md) proposal. It is the "magenta sun number 5" and "magenta sun number 6" problem in the proposal diagram.
+
+The `CanvasToneMapping` API is proposed in [this explainer](canvas-tone-map.md).
+
+## Proposal
+
+Add the following dictionaries for the SMPTE ST 2094-50 color volume transform metadata group.
+
+```idl
+  dictionary SmpteSt2094App5ColorVolumeTransform {
+    required double hdrReferenceWhite;
+    optional SmpteSt2094App5HeadroomAdaptiveToneMap headroomAdaptiveToneMap;
+  }
+  dictionary SmpteSt2094App5HeadroomAdaptiveToneMap {
+    required double baselineHdrHeadroom;
+    required double gainApplicationColorSpacePrimaries[8];
+    required SmpteSt2094App5HeadroomAlternateImage alternateImages[];
+  }
+  dictionary SmpteSt2094App5HeadroomAlternateImage {
+    required double hdrHeadroom;
+    SmpteSt2094App5HeadroomColorGainFunction colorGainFunction;
+  }
+  dictionary SmpteSt2094App5HeadroomColorGainfunction {
+    SmpteSt2094App5HeadroomComponentMixingFunction componentMix;
+    SmpteSt2094App5HeadroomGainCurve gainCurve;
+  }
+  SmpteSt2094App5HeadroomComponentMixingFunction {
+    required double red = 0;
+    required double green = 0;
+    required double blue = 0;
+    required double max = 0;
+    required double min = 0;
+    required double component = 0;
+  }
+  SmpteSt2094App5HeadroomGainCurve {
+    SmpteSt2094App5HeadroomGainCurveControlPoint controlPoints[];
+  }
+  SmpteSt2094App5HeadroomGainCurveControlPoint {
+    required double x;
+    required double y;
+    required double m;
+  }
+```
+
+To the `CanvasToneMappingMode` enum, add an entry to indicate to use SMPTE ST 2094-50.
+
+```idl
+  enum CanvasToneMappingMode {
+      "smpte-st-2094-application-5",
+  }
+```
+
+To the `CanvasToneMapping` dictionary, add an entry for the parameters.
+
+```
+  enum CanvasToneMapping {
+    optional SmpteSt2094App5ColorVolumeTransform smpteSt2094App5ColorVolumeTransform;
+  }
+```
+
+Add a corresponding entry to the [WebCodecs video frame metadata registry](https://www.w3.org/TR/webcodecs-video-frame-metadata-registry/).
+
+## Future directions
+
+This mechanism may be used as the basis for other tone mapping mechanisms, as described in the [canvas tone map](canvas-tone-map#future-directions) explainer.

canvas-tone-map.md

@@ -0,0 +1,158 @@
+# Tone mapping for 2D Canvas
+
+## Introduction
+
+This proposal suggests a mechanism to indicate that the bitmap of a `CanvasRenderingContext2D` or `OffscreenCanvasRenderingContext2D` is HDR, and that it is to be rendered as HDR.
+
+This fits into the [HDR on the web, the big picture](hdr-big-picture.md) proposal. It is the "magenta sun number 5" and "magenta sun number 6" problem in the proposal diagram.
+
+This is a minimal proposal that is required to unify 2D canvas, WebGL, and WebGPU.
+This proposal can be expanded [as indicated in future directions](#future-directions) and in the [SMPTE ST 2094-50 interactions](canvas-smpte-st-2094-50.md) explainer.
+
+### Motivation
+
+The `CanvasRenderingContext2D` or `OffscreenCanvasRenderingContext2D` [output bitmap](https://html.spec.whatwg.org/multipage/canvas.html#output-bitmap) can represent arbitrary color values by using [floating point](https://html.spec.whatwg.org/multipage/canvas.html#dom-canvascolortype-float16) pixel values.
+
+These pixel values may be used to represent higher precision SDR content or HDR content.
+The specification currently does not indicate any way to distinguish between these two uses.
+Nor does the specification have any place to indicate a specific tone mapping mechanism.
+
+### Related work
+
+This problem has existed for WebGPU since its creation, becuase WebGPU has always allowed floating point output buffers via its [`GPUCanvasConfiguration`](https://www.w3.org/TR/webgpu/#dictdef-gpucanvasconfiguration) [`format`](https://www.w3.org/TR/webgpu/#dom-gpucanvasconfiguration-format) parameter.
+
+WebGPU has solved this problem (with an eye towards solving it for 2D canvas and WebGL) via its
+[`GPUCanvasConfiguration`](https://www.w3.org/TR/webgpu/#dictdef-gpucanvasconfiguration)
+[`toneMapping`](https://www.w3.org/TR/webgpu/#dom-gpucanvasconfiguration-tonemapping) parameter.
+
+This parameter is a dictionary of type [`GPUCanvasToneMapping`](https://www.w3.org/TR/webgpu/#dictdef-gpucanvastonemapping).
+At present, the only member is a [`GPUCanvasToneMappingMode`](https://www.w3.org/TR/webgpu/#gpucanvastonemappingmode]) enum.
+
+Please read the documentation of the two values ([`standard`](https://www.w3.org/TR/webgpu/#dom-gpucanvastonemappingmode-standard) and [`extended`](https://www.w3.org/TR/webgpu/#dom-gpucanvastonemappingmode-extended)), since it is precise.
+
+## Proposal
+
+### IDL changes
+
+Rename `GPUCanvasToneMapping` to `CanvasToneMapping`, and move it from the WebGPU specification to the HTML specification.
+The resulting interface would be:
+
+```idl
+  dictionary CanvasToneMapping {
+    CanvasToneMappingMode mode = "standard";
+  }; 
+```
+
+Rename `GPUCanvasToneMappingMode` to `CanvasToneMappingMode`, and move it from the WebGPU specification to the HTML specification.
+The resulting interface would be:
+
+```idl
+  enum CanvasToneMappingMode {
+      "standard",
+      "extended",
+  };
+```
+
+To the `CanvasRenderingContext2DSettings` dictionary, add the following entry:
+
+```idl
+  partial dictionary CanvasRenderingContext2DSettings {
+    CanvasToneMapping toneMapping = {mode:"standard"};
+  }
+```
+
+This indicates that the default behavior is to restrict color values to the standard dynamic range of the target display, which is the current behavior of all browsers.
+
+### Interactions with `dynamic-range-limit`
+
+The `dynamic-range-limit` applies to HTML cavas elements the same as it applies to video and image elements.
+A canvas that specifies to use `"extended"`, but has `dynamic-range-limit:standard` will display the same the canvas would have had it specified `"standard"`, because the "tone mapping" is just to clamp (or project, more accurately) to the available dynamic range.
+For a non-trivial tone mapping, this would not be true.
+
+### Interactions with `ImageBitmap`
+
+When an `ImageBitmap` is created from an `HTMLCanvasElement` or `OffscreenCanvas`
+by any mechanism
+(including
+[`createImageBitmap`](https://html.spec.whatwg.org/multipage/imagebitmap-and-animations.html#dom-createimagebitmap)
+and
+[`transferToImageBitmap`](https://html.spec.whatwg.org/multipage/canvas.html#dom-offscreencanvas-transfertoimagebitmap)),
+the resulting `ImageBitmap` shall preserve the behavior specified by `CanvasToneMapping`.
+
+This includes when it is displayed as an [`ImageBitmapRenderingContext`](https://html.spec.whatwg.org/multipage/canvas.html#the-imagebitmaprenderingcontext-interface),
+and when it is used as a [`CanvasImageSource`](https://html.spec.whatwg.org/multipage/canvas.html#canvasimagesource) or similar.
+
+### Interactions with `VideoFrame`
+
+When a `VideoFrame` is created from an `HTMLCanvasElement` or `OffscreenCanvas`,
+the resulting object should include [metadata](https://www.w3.org/TR/webcodecs-video-frame-metadata-registry/)
+indicating the tone mapping specified.
+
+### Serialization to a file
+
+When a canvas is serialized to a file
+(e.g, by
+[`toDataURL`](https://html.spec.whatwg.org/multipage/canvas.html#dom-canvas-todataurl) 
+or
+[`toBlob`](https://html.spec.whatwg.org/multipage/canvas.html#dom-canvas-toblob)
+or
+[`convertToBlob`](https://html.spec.whatwg.org/multipage/canvas.html#dom-offscreencanvas-converttoblob)),
+the resulting encoded image shall preserve the behavior specified by `CanvasToneMapping`, if supported by the encoded image format.
+
+Canvases that specify `"standard"` tone mapping shall be encoded in a way to maximize the precision in the SDR content.
+If the canvas bitmap uses a floating-point pixel format and has a `PredefinedColorSpace` of `srgb-linear` or `display-p3-linear`,
+but the encoding only supports fixed point pixel formats, then the encoding shall use the transfer function of `srgb` and `display-p3`.
+
+Canvases that specify `"extended"` tone mapping shall be encoded using the HLG (for 8-bit) or PQ (for higher bit depth) transfer function.
+If SMPTE ST 2094-50 HDR metadata can be attached to the resulting ICC profile, then that metadata shall indicate that the tone mapping to perform is a "clamp",
+using the maximum linear HDR headroom value (with is 64 in that specification).
+
+Support for encoding as AVIF and JpegXL should be considered for adding to the relevant APIs.
+
+## Future directions
+
+This explainer has only provided an API for indicating two tone mapping modes. We should consider adding the following modes:
+
+```idl
+  enum CanvasToneMappingMode {
+      "standard",
+      "extended",
+      // The Reinhard tone mapping operator proposed in
+      // Photographic tone reproduction for digital images.
+      // https://doi.org/10.1145/566654.566575
+      "reinhard",
+      // The Reference White tone mapping operator proposed
+      // in ISO 22028-5.
+      "reference-white",
+      // The tone mapping proposed in ITU-R BT.2408 Annex 5.
+      "bt2408",
+      // Tone mapping as specified by SMPTE ST 2094-50.
+      "smpte-st-2094-application-5"
+  };
+```
+
+Along with the following parameters.
+
+```idl
+  dictionary CanvasToneMapping {
+    CanvasToneMappingMode mode = "standard";
+    // Parameter indicating the HDR headroom of the content, which is used
+    // by the "reinhard", "reference-white", and "bt2408" tone mapping
+    // algorithms.
+    optional double linearHDRHeadroom;
+    // Parameters used by "smpte-st-2094-application-5" tone mapping algorithm.
+    optional SmpteSt2094App5ColorVolumeTransform smpteSt2094App5ColorVolumeTransform;
+  }; 
+
+  dictionary SmpteSt2094App5ColorVolumeTransform {
+    // SMPTE ST 2094-50 metadata set.
+  }
+```
+
+The `"reinhard"`, `"reference-white"`, and `"bt2408"` curves can be encoded as
+specific instances of SMPTE ST 2094-50 metadata (which allows them to be attached
+to ICC profiles and videos).
+
+They should be added to the specification along with SMPTE ST 2094-50, to
+allow this simplication.
+