drivers: video: API enhancement

[josuah]

This issue documents the progress of video API changes, pursuing the work on the video API past the Video 4 Zephyr added to Zephyr in #17194.

The main source of inspiration for the video APIs is currently the Linux Kernel v4l2/media subsystem and driver APIs.

Changelog

remote-endpoint is not used by Zephyr drivers. RTIO can be leveraged to replace the many FIFOs that are typically defined in each driver 1, 2, 3 and provide a way to directly connect drivers while also allowing the application to be interleaved in the pipeline. This would act as a splice(2) for Zephyr video drivers, triggered by the devicetree `remote-endpoint

• drivers: video: "remote-endpoint" does nothing #72311 (comment)

[UPDATE: it might be possible to use RTIO on top of the current API instead of replacing it]

---

video_endpoint_id has unclear semantics in some cases (doc fix only) and can be adjusted to allow positive numbers refer an individual endpoint number or address

• drivers: video: introduce semantics for enum endpoint_id #73009

[UPDATE: a documentation fix, mostly]

---

Following this, devicetree macros can be introduced to provide ways to refer a particular endpoint without manually coding the endpoint number from the application.

• include: devicetree: Add port and endpoint macros #80649

---

video_get_caps() fills a struct with all the video caps filled at once. This works well, but makes it more difficult than using an enumerator-style API such as what #72254 proposes which allows drivers to be a bit more generic: a device doing software video processing can filter the format capabilities of its sources this way.

• Video : Add frame interval APIs #72254

---

video_enqueue(): "an addition to the video API to enable streaming partial frames within one video buffer" is missing. A new API is introduced as part of Arducam Mega sensor support.

• drivers: video: video_mcux_smartdma: add SMARTDMA video driver #72827
• drivers: video: Add support for arducam mega #66994 (comment)

---

Introduce a directory for the video drivers, as image sensors, MIPI/DMA controllers [...] are being mixed in one same driver API. A sensor directory was the chosen way and will be merged once the ongoing PR are completed.

• drivers: video: Move the sensors to their dedicated directory #73013

---

Add a drivers/video_*_skeleton.c in the same style as drivers/usb/udc/udc_skeleton.c that can speed-up development and help with understanding the semantics.

• drivers: video: sensors: add a sensor_skeleton.c to speed-up contribution of simple sensors #73867

[UPDATE: proposed as an emulated/fake driver instead]

• video: emul: virtual driver for an imager and RX engine #79482

---

The devicetree can be flattened to enable more uniform processing, such as DT macros used to reduce boilerplate code and homogenize drivers

• dts: flatten video's device tree endpoints definition #72950 (this turns out to not require any change: just conventions)

[UPDATE: no API change needed]

---

VIDEO_BUF_FRAG: Add fragmentation support to allow partial/fragmented frames being returned from video devices.

• drivers: video: Add support for arducam mega #66994 (comment)
• drivers: video: video_mcux_smartdma: add SMARTDMA video driver #72827 (comment)
• usb: device_next: new USB Video Class (UVC) implementation #76798 (comment) > fe3d207

---

video-interfaces.yaml: Introduce common devicetree bindings for all video devices:

• dts: bindings: video: Add common video interface binding #74415

---

<zephyr/drivers/video-controls.h>: Add a flag to the video CIDs that specify variants of current controls to query the kind of result to get: current value (default) or minimum/maximum/default value.

• drivers: video: introduce "GET" sub-operations #78603

---

<zephyr/drivers/video.h>: Introduce types for controls, and add move repeated code from the drivers to a new video framework.

• Implement video control framework #82158

---

<zephyr/drivers/video.h>: Merge video_stream_start() and video_stream_stop() together to reduce code size in the drivers.

• video: Merge driver stream start/stop APIs #85255

---

<zephyr/drivers/video.h>: Add a video_get_stats() API to collect image statistics from drivers.

• drivers: video: image statistics, libcamera-style APIs #85457

---

<zephyr/drivers/video.h>: Remove the video endpoint parameter and instead use a flag to decide between input and output. Multi-endpoint devices can expose multiple struct device *

• video: Drop video endpoint id in video APIs #87070

---

<zephyr/drivers/video.h>: Switch from ad-hoc queueing mechanism and buffer handling to the RTIO implementation:

• drivers: video: switch from custom buffer queues to RTIO #89858

---

<zephyr/drivers/video.h>: Introduce video_set_selection() mirroring the Linux Kernel behavior to crop and scale and pad an image source depending on what the hardware supports. The same API is used by both the sensor and video processing hardware.

• video: selection api (crop/compose) addition (api/shell/sample) + dcmipp implementation #91878

---

<zephyr/drivers/video.h>: Rework the buffering, ongoing discussion about the model to use.

• Video: Rework buffer management #92370

---

m2m First memory-to-memory device introduced. No API change required however this is the first time a video device has both input and output from RAM:

• Video: Display: NXP: Decouple PxP from eLCDIF. Make a new driver for PxP in video m2m category #92366

[josuah]

Extra material to investigate the rationale of the existing APIs, so that their semantics can be kept: https://static.linaro.org/connect/san19/presentations/san19-503.pdf

[josuah]

Thanks to @loicpoulain I rediscovered Gstreamer, and it is great!

The above is also the foundation for a zephyr-shell pipeline, as it is not possible to do it if rewriting the application every time is required. The difficulty, though, is that getting <&phandles> from remote-endpoint at compile-time will not allow to reconnect them later with zephyr-shell.

Maybe it is enough to have "gst-launch pipeline, but all devicetree-made" as a first step...

[teburd]

Right you could treat an rtio_iodev as an endpoint (input or output stream), which is what sensors do today. There's effectively two output streams for sensors, one which is polled, and one which is driven by the sensor itself with an event.

The sensing subsystem provides what amounts to a statically defined pipeline with fanout/fanin possibilities and builds on top of this.

I could imagine video input/output endpoints being similar, though perhaps needing some additional information about image encoding and framesize to describe the metadata.

Image sensors (video) are after all sensors :-)

[josuah]

So let's give it a try! :) I will pick some hardware known to work first and move from here.

To everyone working on a sample or a driver, I wish to avoid disturbing any ongoing effort, so in case I missed an issue, do feel free to ping me about it, I can provide a patch to merge on the PR or delay another PR so that actual driver contribution gets merged first!

[josuah]

Some insightful overview of how libcamera and V4L2 APIs came-up to be:
https://archive.fosdem.org/2022/schedule/event/mobile_camera/
Key takeaways (from the slides):

• Application might want to configure itself each point in the pipeline
• A system should not hard-code use-cases
• There will be vendor-specific controls

Prior work:

• Camera branch #17394