Expose video devices to user #88182

ngphibang · 2025-04-04T23:19:13Z

The video subsystem now exposes a list of (main) video devices to userspace via two APIs:

video_get_dev() : get a video device by index
video_get_devs_num() : get the number of registered video devices

which is similar to the way Linux expose /dev/video0, /dev/video1, etc. sysfs.

With this, we don't need to define a zephyr,camera chosen node anymore. Application can switch at runtime between multiple video devices/pipelines and not restricted to a single pre-defined camera node and despite of the nature of the video devices (HW or SW emulated)

Note that: The video_mdev introduced in this implementation does not only serve for this singple purpose (i.e. expose video devices to user) but also serve as a basis to implement the video buffer management framework that will come in the next PR.

This PR depends on the "Implement video control framework" PR for which I cherry-picked its commit here. The commits to be reviewed are only the last 2 commits of the branch.

Copilot

Copilot reviewed 25 out of 36 changed files in this pull request and generated no comments.

Files not reviewed (11)

boards/arduino/nicla_vision/arduino_nicla_vision_stm32h747xx_m7.dts: Language not supported
boards/espressif/esp32s3_eye/esp32s3_eye_procpu.dts: Language not supported
boards/seeed/xiao_esp32s3/xiao_esp32s3_procpu_sense.dts: Language not supported
boards/shields/dvp_fpc24_mt9m114/dvp_fpc24_mt9m114.overlay: Language not supported
boards/shields/nxp_btb44_ov5640/nxp_btb44_ov5640.overlay: Language not supported
boards/shields/st_b_cams_omv_mb1683/st_b_cams_omv_mb1683.overlay: Language not supported
boards/shields/weact_ov2640_cam_module/weact_ov2640_cam_module.overlay: Language not supported
cmake/linker_script/common/common-ram.cmake: Language not supported
drivers/video/CMakeLists.txt: Language not supported
drivers/video/Kconfig.sw_generator: Language not supported
drivers/video/video.ld: Language not supported

Comments suppressed due to low confidence (1)

drivers/video/ov5640.c:1123

The control case for VIDEO_CID_GAIN appears to be merged into the contrast handling case. Please verify if this mapping is intentional or if a separate analogue gain path (using ov5640_set_ctrl_gain) should be preserved.

return ov5640_set_ctrl_contrast(dev, ctrl->val);

josuah · 2025-04-20T23:09:10Z

How Zephyr devices are currently accessed in Arduino: Use zephyr,camera:

https://github.com/arduino/ArduinoCore-zephyr/blob/2746c8addc5b36faa3610094612db5ddacac1e41/libraries/Camera/src/camera.cpp#L50-L52

How Zephyr devices are currently accessed in MicroPython: Use the device name by string (not affected) by this change

https://github.com/micropython/micropython/blob/master/ports/zephyr/zephyr_device.c#L32

The video_mdev structure is used to determine a video device, usually a DMA or ISP device, as a main device of the video pipeline it belongs to. The main video device represents the whole pipeline to communicate with the application. The video subsystem expose the list of main video devices to userspace via two APIs: - video_get_dev() : get a video device by index - video_get_devs_num() : get the number of registered video devices With this, we don't need to define a zephyr,camera chosen node anymore. Moreover, application can switch between multiple video devices/pipelines and not restricted to a single pre-defined camera node. Signed-off-by: Phi Bang Nguyen <phibang.nguyen@nxp.com>

…mera presents" This reverts commit 2ff984a. As of now, "zephyr,camera" chosen node is dropped so the video sw generator can be built and used beside any other video hardwares. Signed-off-by: Phi Bang Nguyen <phibang.nguyen@nxp.com>

ngphibang · 2025-04-21T19:18:05Z

The PR is independent and enough to accomplish its purpose of exposing video device to user but convert it to draft to favor #88780 and also to add more things to video_mdev struct to show its full usage for buffer management.

josuah

It seems like solving problems and reduce the boilerplate.

I suspect the concept of "software interface backed by hardware or software" shows-up at several places through Zephyr:

In case something generic like "interface selection and configuration" is introduced, the current implementation may be able to plug to it.

Glad to see this merged. :)
Waiting your feedback before giving it a LGTM

josuah · 2025-05-11T13:29:32Z

drivers/video/video_device.h

+struct video_mdev {
+	const struct device *dev;
+};
+


How about video_interface to communicate that this is the part of the video chain that an application interacts with. Hopefully reducing the jargon required to understand V4Z a little.

An alternative is to use STRUCT_SECTION_ITERABLE_ALTERNATE so that defining a wrapper struct is not required. Instead, a struct device is used, but with a dedicated section name such as video_mdev or video_interface:

zephyr/include/zephyr/sys/iterable_sections.h

Lines 180 to 189 in c35bb0d

/**

* @brief Defines a new element of alternate data type for an iterable section.

*

* @details

* Special variant of STRUCT_SECTION_ITERABLE(), for placing alternate

* data types within the iterable section of a specific data type. The

* data type sizes and semantics must be equivalent!

*/

#define STRUCT_SECTION_ITERABLE_ALTERNATE(secname, struct_type, varname) \

TYPE_SECTION_ITERABLE(struct struct_type, varname, secname, varname)

For instance here is how USB submits the same struct type but with different names for each USB speed:

zephyr/include/zephyr/usb/usbd.h

Lines 745 to 752 in c35bb0d

static STRUCT_SECTION_ITERABLE_ALTERNATE( \

usbd_class_fs, usbd_class_node, class_name##_fs) = { \

.c_data = &class_name, \

}; \

static STRUCT_SECTION_ITERABLE_ALTERNATE( \

usbd_class_hs, usbd_class_node, class_name##_hs) = { \

.c_data = &class_name, \

}

Both approach work depending on your goals, i.e. plan extra fields in the future or not.

josuah · 2025-05-11T13:35:25Z

drivers/video/video_device.h

+#define VIDEO_MDEV_DEFINE(name, device)                                                            \
+	static STRUCT_SECTION_ITERABLE(video_mdev, name_##mdev) = {                                \
+		.dev = device,                                                                     \
+	}
+


To reduce the boilerplate, it might be possible to combine the two declarations in one:

Suggested change

#define VIDEO_MDEV_DEFINE(name, device) \

static STRUCT_SECTION_ITERABLE(video_mdev, name_##mdev) = { \

.dev = device, \

}

#define VIDEO_INTERFACE_DEFINE(name, device, source) \

VIDEO_DEVICE_DEFINE(name, device, source); \

static STRUCT_SECTION_ITERABLE(video_interface, name_##interface) = { \

.dev = device, \

}

Then the user can select VIDEO_DEVICE_DEFINE() if it is an intermediate device, and VIDEO_INTERFACE_DEFINE() if it is a media I/O device to define both at once.

This also prevents to define a VIDEO_INTERFACE_DEFINE() without a VIDEO_DEVICE_DEFINE() by mistake.

josuah · 2025-05-11T13:37:56Z

samples/drivers/video/capture/src/main.c

+	/* Get the 1st video HW available, otherwise fallbacks to video sw generator */
+	for (int j = 0; j < vdevs_num; j++) {
+		video_dev = video_get_dev(j);
+		if (device_is_ready(video_dev) && (vdevs_num == 1 || video_dev != vsg)) {


Given that device_is_ready() == false when a device init() returns an error, how about a LOW_WRN("video device %s failed to initialize", video_dev->name) to help user notice the issue?

samples/drivers/video/capture/src/main.c

josuah · 2025-05-11T13:54:32Z

samples/drivers/video/capture_to_lvgl/src/main.c

-	if (!device_is_ready(video_dev)) {
-		LOG_ERR("%s device is not ready", video_dev->name);
-		return 0;
+	/* Get the 1st video HW available, otherwise fallbacks to video sw generator */


It seems like this PR implements a fallback mechanism already, so is video_dev != vsg really needed?

vsg is also registered and listed by the video framework. So, if it is listed at first place and in case we have multiple video devices, we should continue the loop to get the 1st real hw video device, otherwise we stop the loop because we found the real hw device or we only have one video device (whatever it is sw or hw, we have no choice).

This makes sense. Great example of how to do filtering.

if it is listed at first place

One How does an user control which device is at the first place? I think it is alphabetical sorting.

Could there be 2 devices, with the first being the VIDEO_SW_GENERATOR, and then the VIDEO_SW_GENERATOR would be selected instead of the real camera?

github-actions bot added platform: ESP32 Espressif ESP32 platform: NXP Drivers NXP Semiconductors, drivers area: Samples Samples area: Video Video subsystem platform: STM32 ST Micro STM32 area: Shields Shields (add-on boards) area: Build System labels Apr 4, 2025

github-actions bot requested review from avisconti, dbaluta, decsny, dleach02, erwango, jfischer-no, josuah, kartben, loicpoulain, LucasTambor, manuargue, mmahadevan108, nashif, Raymond0225 and sylvioalves April 4, 2025 23:20

github-actions bot assigned kartben Apr 4, 2025

kartben requested review from Copilot and removed request for avisconti April 5, 2025 14:58

Copilot AI reviewed Apr 5, 2025

View reviewed changes

kartben removed their assignment Apr 7, 2025

decsny removed their request for review April 14, 2025 16:06

ngphibang mentioned this pull request Apr 17, 2025

video: Drop video endpoint id in video APIs #87070

Merged

josuah mentioned this pull request Apr 18, 2025

drivers: video: CONFIG_VIDEO_SW_GENERATOR -> zephyr,video-sw-generator #88780

Merged

ngphibang marked this pull request as draft April 21, 2025 19:06

ngphibang added 2 commits April 21, 2025 21:09

ngphibang force-pushed the expose_video_device_to_user branch from dcdb8ef to 7521f41 Compare April 21, 2025 19:10

ngphibang mentioned this pull request Apr 22, 2025

usb: device_next: new USB Video Class (UVC) implementation #76798

Draft

ngphibang mentioned this pull request May 8, 2025

drivers: video: Add the video shell #88566

Merged

josuah reviewed May 11, 2025

View reviewed changes

josuah mentioned this pull request Jun 2, 2025

drivers: video: samples: improve the video-sw-generator doc #90937

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Expose video devices to user #88182

Expose video devices to user #88182

ngphibang commented Apr 4, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

josuah commented Apr 20, 2025 •

edited

Loading

Uh oh!

ngphibang commented Apr 21, 2025

Uh oh!

josuah left a comment

Uh oh!

josuah May 11, 2025

Uh oh!

josuah May 11, 2025

Uh oh!

josuah May 11, 2025

Uh oh!

josuah May 11, 2025

Uh oh!

Uh oh!

josuah May 11, 2025

Uh oh!

ngphibang May 11, 2025

Uh oh!

josuah May 11, 2025 •

edited

Loading

Uh oh!

Uh oh!

	/**
	* @brief Defines a new element of alternate data type for an iterable section.
	*
	* @details
	* Special variant of STRUCT_SECTION_ITERABLE(), for placing alternate
	* data types within the iterable section of a specific data type. The
	* data type sizes and semantics must be equivalent!
	*/
	#define STRUCT_SECTION_ITERABLE_ALTERNATE(secname, struct_type, varname) \
	TYPE_SECTION_ITERABLE(struct struct_type, varname, secname, varname)

	static STRUCT_SECTION_ITERABLE_ALTERNATE( \
	usbd_class_fs, usbd_class_node, class_name##_fs) = { \
	.c_data = &class_name, \
	}; \
	static STRUCT_SECTION_ITERABLE_ALTERNATE( \
	usbd_class_hs, usbd_class_node, class_name##_hs) = { \
	.c_data = &class_name, \
	}

Expose video devices to user #88182

Are you sure you want to change the base?

Expose video devices to user #88182

Conversation

ngphibang commented Apr 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

josuah commented Apr 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ngphibang commented Apr 21, 2025

Uh oh!

josuah left a comment

Choose a reason for hiding this comment

Uh oh!

josuah May 11, 2025

Choose a reason for hiding this comment

Uh oh!

josuah May 11, 2025

Choose a reason for hiding this comment

Uh oh!

josuah May 11, 2025

Choose a reason for hiding this comment

Uh oh!

josuah May 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

josuah May 11, 2025

Choose a reason for hiding this comment

Uh oh!

ngphibang May 11, 2025

Choose a reason for hiding this comment

Uh oh!

josuah May 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ngphibang commented Apr 4, 2025 •

edited

Loading

josuah commented Apr 20, 2025 •

edited

Loading

josuah May 11, 2025 •

edited

Loading