Zero-copy: introduce a way to letterbox the BlitRequest results

Currently, CopyOutputRequest API enables its callers to obtain a copy
of the contents produced by a render pass. BlitRequest allows those
results to be placed in a specific region of caller-provided textures,
without modifying contents outside of that region. This is an
almost-ideal situation for FrameSinkVideoCapturerImpl, with an
exception that when it produces a VideoFrame, it may be forced to
letterbox part of the frame. To avoid doing this work in the capturer
(by memory-mapping the GpuMemoryBuffer), a letterbox region is
introduced to BlitRequest.

Changes:
- Introduce `BlitRequest::letterbox_region()` that specifies the
  region of the caller-provided textures outside of which everything
  will be filled with black.
- `FrameSinkVideoCaprturerImpl` takes advantage of newly introduced
  capability on `BlitRequest`. Letterboxing only happens in the
  capturer for non-GMB-backed VideoFrames, and only if there are parts
  of a frame that we have not already written to.
- `skia::BlitRGBAToYUVA()` now exposes additional parameter to allow
  letterboxing of the destination image.
- Test changes in SkiaReadbackPixeltest to account for new parameter
  to `BlitRequest`.
- Minor: `RenderableGpuMemoryBufferVideoFramePool` now tags
  GpuMemoryBuffers with the color space.
- Minor cleanup in
  `SkiaOutputSurfaceImplOnGpu::ImportSurfacesForNV12Planes()` - rename
  variables, add/expand comments, add DVLOG(3)s to facilitate
  future investigations.
- Minor cleanup in `FrameSinkVideoCapturerImpl` - comments, DVLOGs,
  helpers.

Bug: 1310411
Change-Id: I292ab78b9aabc29e23eb1489ea1ca8c9752b88eb
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3617204
Reviewed-by: Jordan Bayles <jophba@chromium.org>
Reviewed-by: Dan Sanders <sandersd@chromium.org>
Reviewed-by: Michael Ludwig <michaelludwig@google.com>
Commit-Queue: Piotr Bialecki <bialpio@chromium.org>
Reviewed-by: Kyle Charbonneau <kylechar@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1003391}

components/viz/common/frame_sinks/README.md

@@ -56,8 +56,8 @@ properties:
    Note that all coordinates are constrained to be integer values, to avoid
    introducing alignment, rounding or other "fuzz" issues.
 
- * Result format: An RGBA-interleaved bitmap (SkBitmap) or I420 Y+U+V image
-   planes.
+ * Result format: An RGBA-interleaved bitmap (SkBitmap), I420 Y+U+V image
+   planes, or NV12 Y+UV image planes.
 
 For efficient video capture, the above are used as follows: An issuer of
 CopyOutputRequests "locks into" a target area within the Surface (usually the

components/viz/common/frame_sinks/blit_request.cc

@@ -31,9 +31,11 @@ std::string BlendBitmap::ToString() const {
 
 BlitRequest::BlitRequest(
     const gfx::Point& destination_region_offset,
+    LetterboxingBehavior letterboxing_behavior,
     const std::array<gpu::MailboxHolder, CopyOutputResult::kMaxPlanes>&
         mailboxes)
     : destination_region_offset_(destination_region_offset),
+      letterboxing_behavior_(letterboxing_behavior),
       mailboxes_(mailboxes) {}
 
 BlitRequest::BlitRequest(BlitRequest&& other) = default;

components/viz/common/frame_sinks/blit_request.h

@@ -53,13 +53,24 @@ class VIZ_COMMON_EXPORT BlendBitmap {
   sk_sp<SkImage> image_;
 };
 
+// Enum used to specify letteboxing behavior for a BlitRequest.
+enum class LetterboxingBehavior {
+  // No letterboxing is needed - only the destination region will be written
+  // into by the handler of CopyOutputRequest.
+  kDoNotLetterbox,
+  // Letterboxing is needed - everything outside of the destination region
+  // will be filled with black by the handler of CopyOutputRequest.
+  kLetterbox
+};
+
 // Structure describing a blit operation that can be appended to
 // `CopyOutputRequest` if the callers want to place the results of the operation
 // in textures that they own.
 class VIZ_COMMON_EXPORT BlitRequest {
  public:
   explicit BlitRequest(
       const gfx::Point& destination_region_offset,
+      LetterboxingBehavior letterboxing_behavior,
       const std::array<gpu::MailboxHolder, CopyOutputResult::kMaxPlanes>&
           mailboxes);
 
@@ -74,6 +85,10 @@ class VIZ_COMMON_EXPORT BlitRequest {
     return destination_region_offset_;
   }
 
+  LetterboxingBehavior letterboxing_behavior() const {
+    return letterboxing_behavior_;
+  }
+
   const std::array<gpu::MailboxHolder, CopyOutputResult::kMaxPlanes>&
   mailboxes() const {
     return mailboxes_;
@@ -106,6 +121,9 @@ class VIZ_COMMON_EXPORT BlitRequest {
   // images.
   gfx::Point destination_region_offset_;
 
+  // Specifies the letterboxing behavior of this request.
+  LetterboxingBehavior letterboxing_behavior_;
+
   // Mailboxes with planes that will be populated.
   // The textures can (but don't have to be) backed by
   // a GpuMemoryBuffer. The pixel format of the request determines

components/viz/common/frame_sinks/copy_output_request.cc

@@ -107,6 +107,7 @@ void CopyOutputRequest::set_blit_request(BlitRequest blit_request) {
   DCHECK(!blit_request_);
   DCHECK_EQ(result_destination(), ResultDestination::kNativeTextures);
   DCHECK_EQ(result_format(), ResultFormat::NV12_PLANES);
+  DCHECK(has_result_selection());
 
   // Destination region must start at an even offset for NV12 results:
   DCHECK_EQ(blit_request.destination_region_offset().x() % 2, 0);

components/viz/common/frame_sinks/copy_output_request.h

@@ -116,20 +116,30 @@ class VIZ_COMMON_EXPORT CopyOutputRequest {
   // Optionally specify that only a portion of the result be generated. The
   // selection rect will be clamped to the result bounds, which always starts at
   // 0,0 and spans the post-scaling size of the copy area (see set_area()
-  // above). Only RGBA format supports odd-sized result selection.
+  // above). Only RGBA format supports odd-sized result selection. Can only be
+  // called before blit request was set on the copy request.
   void set_result_selection(const gfx::Rect& selection) {
     DCHECK(result_format_ == ResultFormat::RGBA ||
            (selection.width() % 2 == 0 && selection.height() % 2 == 0))
         << "CopyOutputRequest supports odd-sized result_selection() only for "
            "RGBA!";
+    DCHECK(!has_blit_request());
     result_selection_ = selection;
   }
   bool has_result_selection() const { return result_selection_.has_value(); }
   const gfx::Rect& result_selection() const { return *result_selection_; }
 
   // Requests that the region copied by the CopyOutputRequest be blitted into
   // the caller's textures. Can be called only for CopyOutputRequests that
-  // target native textures.
+  // target native textures. Requires that result selection was set, in which
+  // case the caller's textures will be populated with the results of the
+  // copy request. The region in the caller's textures that will be populated
+  // is specified by `gfx::Rect(blit_request.destination_region_offset(),
+  // result_selection().size())`. If blit request is configured to perform
+  // letterboxing, all contents outside of that region will be overwritten with
+  // black, otherwise they will be unchanged. If the copy request's result would
+  // be smaller than `result_selection().size()`, the request will fail (i.e.
+  // empty result will be sent).
   void set_blit_request(BlitRequest blit_request);
   bool has_blit_request() const { return blit_request_.has_value(); }
   const BlitRequest& blit_request() const { return *blit_request_; }

components/viz/service/display/skia_readback_pixeltest.cc

@@ -17,6 +17,7 @@
 #include "cc/test/pixel_test.h"
 #include "cc/test/pixel_test_utils.h"
 #include "cc/test/resource_provider_test_utils.h"
+#include "components/viz/common/frame_sinks/blit_request.h"
 #include "components/viz/common/frame_sinks/copy_output_request.h"
 #include "components/viz/common/frame_sinks/copy_output_result.h"
 #include "components/viz/common/frame_sinks/copy_output_util.h"
@@ -572,7 +573,8 @@ GTEST_ALLOW_UNINSTANTIATED_PARAMETERIZED_TEST(SkiaReadbackPixelTestNV12);
 
 class SkiaReadbackPixelTestNV12WithBlit
     : public SkiaReadbackPixelTest,
-      public testing::WithParamInterface<bool> {
+      public testing::WithParamInterface<
+          std::tuple<bool, LetterboxingBehavior>> {
  public:
   CopyOutputResult::Destination RequestDestination() const {
     return CopyOutputResult::Destination::kNativeTextures;
@@ -583,7 +585,11 @@ class SkiaReadbackPixelTestNV12WithBlit
   }
 
   void SetUp() override {
-    SkiaReadbackPixelTest::SetUpReadbackPixeltest(GetParam());
+    SkiaReadbackPixelTest::SetUpReadbackPixeltest(std::get<0>(GetParam()));
+  }
+
+  LetterboxingBehavior GetLetterboxingBehavior() const {
+    return std::get<1>(GetParam());
   }
 };
 
@@ -625,8 +631,7 @@ TEST_P(SkiaReadbackPixelTestNV12WithBlit, ExecutesCopyRequestWithBlit) {
       << " The test case expects the blit region's origin to be even for NV12 "
          "blit requests";
 
-  const SkColor rgba_red = SkColorSetARGB(0xff, 0xff, 0, 0);
-  const SkColor yuv_red = GLScalerTestUtil::ConvertRGBAColorToYUV(rgba_red);
+  const SkColor yuv_red = GLScalerTestUtil::ConvertRGBAColorToYUV(SK_ColorRED);
 
   const std::vector<uint8_t> luma_pattern = {
       static_cast<uint8_t>(SkColorGetR(yuv_red))};
@@ -669,8 +674,9 @@ TEST_P(SkiaReadbackPixelTestNV12WithBlit, ExecutesCopyRequestWithBlit) {
 
         request.set_result_selection(result_selection);
 
-        request.set_blit_request(
-            BlitRequest(destination_subregion.origin(), mailboxes));
+        request.set_blit_request(BlitRequest(destination_subregion.origin(),
+                                             GetLetterboxingBehavior(),
+                                             mailboxes));
       }));
 
   // Check that a result was produced and is of the expected rect/size.
@@ -716,7 +722,18 @@ TEST_P(SkiaReadbackPixelTestNV12WithBlit, ExecutesCopyRequestWithBlit) {
   // The textures that we passed in to BlitRequest contained NV12 plane data for
   // an all-red image, let's re-create such a bitmap:
   SkBitmap expected = GLScalerTestUtil::AllocateRGBABitmap(source_size);
-  expected.eraseColor(rgba_red);
+
+  if (GetLetterboxingBehavior() == LetterboxingBehavior::kLetterbox) {
+    // We have requested the results to be letterboxed, so everything that
+    // CopyOutputRequest is not populating w/ render pass contents should be
+    // black:
+    expected.eraseColor(SK_ColorBLACK);
+  } else {
+    // We have requested the results to not be letterboxed, so everything that
+    // CopyOutputRequest is not populating w/ render pass will have original
+    // contents (red in our case):
+    expected.eraseColor(SK_ColorRED);
+  }
 
   // Blit request should "stitch" the pixels from the source image into a
   // sub-region of caller-provided texture - let's write our expected pixels
@@ -732,10 +749,12 @@ TEST_P(SkiaReadbackPixelTestNV12WithBlit, ExecutesCopyRequestWithBlit) {
 }
 
 #if !BUILDFLAG(IS_ANDROID) || !defined(ARCH_CPU_X86_FAMILY)
-INSTANTIATE_TEST_SUITE_P(,
-                         SkiaReadbackPixelTestNV12WithBlit,
-                         // Result scaling: Scale by half?
-                         testing::Values(true, false));
+INSTANTIATE_TEST_SUITE_P(
+    ,
+    SkiaReadbackPixelTestNV12WithBlit,
+    testing::Combine(testing::Bool(),  // Result scaling: Scale by half?
+                     testing::Values(LetterboxingBehavior::kDoNotLetterbox,
+                                     LetterboxingBehavior::kLetterbox)));
 #else
 // Don't instantiate the NV12 tests when run on Android emulator, they won't
 // work since the SkiaRenderer currently does not support CopyOutputRequests

components/viz/service/display_embedder/skia_output_surface_impl_on_gpu.cc

@@ -849,8 +849,8 @@ bool SkiaOutputSurfaceImplOnGpu::ImportSurfacesForNV12Planes(
   for (size_t i = 0; i < CopyOutputResult::kNV12MaxPlanes; ++i) {
     const gpu::MailboxHolder& mailbox_holder = blit_request.mailbox(i);
 
-    // Should never happen, maiboxes are validated when setting blit request on
-    // a CopyOutputResult.
+    // Should never happen, mailboxes are validated when setting blit request on
+    // a CopyOutputResult and we only access `kNV12MaxPlanes` mailboxes.
     DCHECK(!mailbox_holder.mailbox.IsZero());
 
     PlaneAccessData& plane_data = plane_access_datas[i];
@@ -912,6 +912,9 @@ void SkiaOutputSurfaceImplOnGpu::CopyOutputNV12(
              CopyOutputRequest::ResultDestination::kNativeTextures)
       << "Only CopyOutputRequests that hand out native textures support blit "
          "requests!";
+  DCHECK(!request->has_blit_request() || request->has_result_selection())
+      << "Only CopyOutputRequests that specify result selection support blit "
+         "requests!";
 
   // Overview:
   // 1. Try to create surfaces for NV12 planes (we know the needed size in
@@ -934,16 +937,29 @@ void SkiaOutputSurfaceImplOnGpu::CopyOutputNV12(
   // copied, as well as area, scaling & result_selection of the `request`.
   // This represents the size of the intermediate texture that will be then
   // blitted to the destination textures.
-  const gfx::Size destination_size = geometry.result_selection.size();
+  const gfx::Size intermediate_dst_size = geometry.result_selection.size();
 
   std::array<PlaneAccessData, CopyOutputResult::kNV12MaxPlanes>
       plane_access_datas;
 
   SkYUVAInfo yuva_info;
 
-  bool destination_surfaces_created = false;
+  bool destination_surfaces_ready = false;
   if (request->has_blit_request()) {
-    destination_surfaces_created = ImportSurfacesForNV12Planes(
+    if (request->result_selection().size() != intermediate_dst_size) {
+      DLOG(WARNING)
+          << __func__
+          << ": result selection is different than render pass output, "
+             "geometry="
+          << geometry.ToString() << ", request=" << request->ToString();
+      // Send empty result, we have a blit request that asks for a different
+      // size than what we have available - the behavior in this case is
+      // currently unspecified as we'd have to leave parts of the caller's
+      // region unpopulated.
+      return;
+    }
+
+    destination_surfaces_ready = ImportSurfacesForNV12Planes(
         request->blit_request(), plane_access_datas);
 
     // The entire destination image size is the same as the size of the luma
@@ -954,7 +970,8 @@ void SkiaOutputSurfaceImplOnGpu::CopyOutputNV12(
 
     // Check if the destination will fit in the blit target:
     const gfx::Rect blit_destination_rect(
-        request->blit_request().destination_region_offset(), destination_size);
+        request->blit_request().destination_region_offset(),
+        intermediate_dst_size);
     const gfx::Rect blit_target_image_rect(
         gfx::SkISizeToSize(plane_access_datas[0].size));
 
@@ -964,24 +981,25 @@ void SkiaOutputSurfaceImplOnGpu::CopyOutputNV12(
       return;
     }
   } else {
-    yuva_info = SkYUVAInfo(
-        gfx::SizeToSkISize(destination_size), SkYUVAInfo::PlaneConfig::kY_UV,
-        SkYUVAInfo::Subsampling::k420, kRec709_Limited_SkYUVColorSpace);
+    yuva_info = SkYUVAInfo(gfx::SizeToSkISize(intermediate_dst_size),
+                           SkYUVAInfo::PlaneConfig::kY_UV,
+                           SkYUVAInfo::Subsampling::k420,
+                           kRec709_Limited_SkYUVColorSpace);
 
-    destination_surfaces_created =
+    destination_surfaces_ready =
         CreateSurfacesForNV12Planes(yuva_info, color_space, plane_access_datas);
   }
 
-  if (!destination_surfaces_created) {
-    DVLOG(1) << "failed to create destination surfaces";
+  if (!destination_surfaces_ready) {
+    DVLOG(1) << "failed to create / import destination surfaces";
     // Send empty result.
     return;
   }
 
   // Create a destination for the scaled & clipped result:
-  auto representation = CreateSharedImageRepresentationSkia(
-      ResourceFormat::RGBA_8888, destination_size, color_space);
-  if (!representation) {
+  auto intermediate_representation = CreateSharedImageRepresentationSkia(
+      ResourceFormat::RGBA_8888, intermediate_dst_size, color_space);
+  if (!intermediate_representation) {
     DVLOG(1) << "failed to create shared image representation for the "
                 "intermediate surface";
     // Send empty result.
@@ -992,9 +1010,11 @@ void SkiaOutputSurfaceImplOnGpu::CopyOutputNV12(
   std::vector<GrBackendSemaphore> begin_semaphores;
   std::vector<GrBackendSemaphore> end_semaphores;
 
-  auto scoped_write = representation->BeginScopedWriteAccess(
-      /*final_msaa_count=*/1, surface_props, &begin_semaphores, &end_semaphores,
-      gpu::SharedImageRepresentation::AllowUnclearedAccess::kYes);
+  auto intermediate_scoped_write =
+      intermediate_representation->BeginScopedWriteAccess(
+          /*final_msaa_count=*/1, surface_props, &begin_semaphores,
+          &end_semaphores,
+          gpu::SharedImageRepresentation::AllowUnclearedAccess::kYes);
 
   absl::optional<SkVector> scaling;
   if (request->is_scaled()) {
@@ -1004,21 +1024,22 @@ void SkiaOutputSurfaceImplOnGpu::CopyOutputNV12(
                                  request->scale_from().y());
   }
 
-  scoped_write->surface()->wait(begin_semaphores.size(),
-                                begin_semaphores.data());
+  intermediate_scoped_write->surface()->wait(begin_semaphores.size(),
+                                             begin_semaphores.data());
 
   RenderSurface(surface, src_rect, scaling,
                 is_downscale_or_identity_in_both_dimensions,
-                scoped_write->surface());
+                intermediate_scoped_write->surface());
 
   if (request->has_blit_request()) {
-    BlendBitmapOverlays(scoped_write->surface()->getCanvas(),
+    BlendBitmapOverlays(intermediate_scoped_write->surface()->getCanvas(),
                         request->blit_request());
   }
 
-  auto source_image = scoped_write->surface()->makeImageSnapshot();
-  if (!source_image) {
-    DLOG(ERROR) << "failed to retrieve source_image.";
+  auto intermediate_image =
+      intermediate_scoped_write->surface()->makeImageSnapshot();
+  if (!intermediate_image) {
+    DLOG(ERROR) << "failed to retrieve `intermediate_image`.";
     return;
   }
 
@@ -1030,19 +1051,27 @@ void SkiaOutputSurfaceImplOnGpu::CopyOutputNV12(
       plane_access_datas[1].scoped_write->surface(), nullptr, nullptr};
 
   // The region to be populated in caller's textures is derived from blit
-  // request's |destination_region_offset|, and from COR's |result_selection|.
-  // If we have a blit request, use it. Otherwise, use an
+  // request's |destination_region_offset()|, and from COR's
+  // |result_selection()|. If we have a blit request, use it. Otherwise, use an
   // empty rect (which means that entire image will be used as the target of the
   // blit - this will not result in rescaling since w/o blit request present,
-  // the image size matches the |result_selection|).
-  SkRect dst_region =
+  // the destination image size matches the |geometry.result_selection|).
+  const SkRect dst_region =
       request->has_blit_request()
           ? gfx::RectToSkRect(
                 gfx::Rect(request->blit_request().destination_region_offset(),
-                          destination_size))
+                          intermediate_dst_size))
           : SkRect::MakeEmpty();
-  skia::BlitRGBAToYUVA(source_image.get(), plane_surfaces.data(), yuva_info,
-                       dst_region);
+
+  // We should clear destination if BlitRequest asked to letterbox everything
+  // outside of intended destination region:
+  const bool clear_destination =
+      request->has_blit_request()
+          ? request->blit_request().letterboxing_behavior() ==
+                LetterboxingBehavior::kLetterbox
+          : false;
+  skia::BlitRGBAToYUVA(intermediate_image.get(), plane_surfaces.data(),
+                       yuva_info, dst_region, clear_destination);
 
   bool should_submit = false;
 
@@ -1059,12 +1088,12 @@ void SkiaOutputSurfaceImplOnGpu::CopyOutputNV12(
     }
   }
 
-  representation->SetCleared();
+  intermediate_representation->SetCleared();
 
   should_submit |= !end_semaphores.empty();
 
-  if (!FlushSurface(scoped_write->surface(), end_semaphores,
-                    scoped_write->TakeEndState())) {
+  if (!FlushSurface(intermediate_scoped_write->surface(), end_semaphores,
+                    intermediate_scoped_write->TakeEndState())) {
     // TODO(penghuang): handle vulkan device lost.
     FailedSkiaFlush("CopyOutputNV12 dest_surface->flush()");
     return;