-
Notifications
You must be signed in to change notification settings - Fork 7.7k
VIDEO: GC2145/stm32_dcmi support for CROP/Snapshot #93797
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
558b218
to
aa416e3
Compare
@josuah @mjs513 @dkalowsk @iabdalkader and all: As @josuah mentioned in my previous PR:
Which makes sense: But now wondering about a few details. In my own test case app I have:
And the CONF file has:
But if I was not using my updated fmts, which allows more resolutions, I would have done FRAME_WIDTH=640 and HEIGHT=480 So now assume:
With this, the call to video_set_format would have a width=640 and height= 480 Note: If you now (first commit) call video_get_selection with VIDEO_SEL_TGT_NATIVE_SIZE, it will return 800x600 So now to do step 2) to crop it to 480x320 - Currently I ignore the passed in top and left in the crop rectangle, but that So with this setup, I would expect on the TGT_CROP that I should be able to pass in rectangles in the range: But in this case: should step 1) have set the top=0, left=0 or should it instead of set it to 80,60? Thanks |
Having problems with this PR first commit on getting signoff to be accepted: I actually copy/pasted the signoff line from previous PRS which worked then but not on this one?
All of which failed like: -- Run compliance checks on patch series (PR): Identity.txt#L0See https://docs.zephyrproject.org/latest/contribute/guidelines.html#commit-guidelines for more details 52ea30be81e17759c1725a3ef7e7c2c64d6ed5c0: Signed-off-by line (Signed-off-by: Kurt Eckhardt ) does not follow the syntax: First Last .52ea30be81e17759c1725a3ef7e7c2c64d6ed5c0: Signed-off-by line (Signed-off-by: Kurt Eckhardt kurte@rockisland.com) does not follow the syntax: First Last . |
f1d99dd
to
acb2521
Compare
More mumbling to self ;)
Reworking: Currently I have the crop code recalculate most of the window and crop registers. Will instead have it As the setting the crop updates the fmt structure:
Why? Because the buffer management code requires the buffers to be that size:
So for example if I setup for the GC2145 camera to output 480x320 the buffer size needed is: 307200 bytes This also effects the range of crop LEFT and TOP to fit the 480x320 within the range of 640x480... |
3b43aca
to
019f84b
Compare
Note: this all uses the Zephyr updates from my PR zephyrproject-rtos/zephyr#93797 Which I added to the STM dcmi driver the ability to have the camera work in snapshot mode instead of in continuous video mode. This allows for example that we start the camera it grabs a frame and stops, we then take the buffer and process it, and repeat this. This helps minimize how much the SDRAM gets used concurrently. In addition, I added to the VIDEO_STM32_DCMI and GC2145 the ability to use some of the new set_selection and get_selection code that was added for the DCMIPP. In particular the DCMI simply forwards these messages to the camera if it defined thise apis... And with this it allows you to setup a viewport into the frame. For example: You can setup the frame on the GC2145 to be 800x600 and then define a view port to be 480x320 to fill an ST7796/ILI9486 tft display or you could do it 400x240 to half fill the GIGA display. You can also move that view port around within the frame (pan) I have examples that do this on Portenta H7 on the ST7796 display and another one that does this on the GIGA display shield. Still WIP as we probably need to refine the APIS and the like
0cb0ea4
to
74b3afe
Compare
Thanks @KurtE. Before going into the review of the commits, I am wondering about the 2 new API get_snapshot_mode and set_snapshot_mode especially since you indicate that now the driver is now updated in order to work with only a single buffer. |
Thanks @avolmat-st - I am in a high percentage agreement with you, that we probably don't need those two APIs in the zephyr world, as when you setup your application, you probably know how you want it to work. Where it becomes less clear to me, is when you then add on the ArduinoCore-zephyr layer on top of it. As it is currently setup, the camera configuration setup is defined as part of building the "bootloader". Which if you use But probably that can be controlled by, how many buffers are defined as MAX and/or if at least two were defined So I believe these could be removed. Not sure if zephyr apps might want to know if the captures are done continuous or Note: I believe in most cases on these STM32ST... Arduino boards, they will likely always run in snapshot mode. So if you prefer I can remove these APIS. Thanks |
Yes, I think we don't need an API for this too. In case you really need to have some control on it on the application side, we can make it vendor private control ID, something like ?
|
Added support to video_stm32_dcmi for the new set and get selection. These implementations simply forward the message to the underlying camera object if they support these messages. Also added support for a snapshot mode instead of always using continuous capture mode. Tried to make it semi-transparent when you desire it to be The stm32_dcmi code now also allows you to work with only one buffer. This will force it into snapshot mode. There is also new calls added to the api for: video_get_snapshot_mode and video_set_snapshot_mode. That allows you to set it with more than one buffer and query what mode you are in. GC2145 was updated first to try out these changes. The camera now allows me to follow the call order that @josuah mentioned in another pr/issue. With this driver I also updated it to allow any resolution from the displays min to max limits. static const struct video_format_cap fmts[] = { GC2145_VIDEO_FORMAT_CAP_HL(128, 1600, 128, 1200, VIDEO_PIX_FMT_RGB565), GC2145_VIDEO_FORMAT_CAP_HL(128, 1600, 128, 1200, VIDEO_PIX_FMT_YUYV), It also still retains the other resolutions as some software does not check if it is within a range but only against the min value. When the resolution is set, it computes the scale factor. If you then later call set_crop, the same code is used except it uses the ratios computed from the set_resolution. Currently it clamps the ratio to a max of 3 as some other drivers limit it saying it helps with frame rates. With these changes: I was able to setup a test app, for the Arduino Nicla vision and send out a 480x320 image over USB. Video: GC2145 update set Format/Crop as mentioned As such if you passed in 640x480 on set_format, that is what you are limited to. So VIDEO_SEL_TGT_NATIVE_SIZE will return 640x480, so I need to save that away and/or grab it from registers. As the setting the crop updates the fmt structure: Also limited ratio in set resolution to 3 as it was mentioned in other implementations to help with frame rate. Signed-off-by: Kurt Eckhardt <kurte@rockisland.com>
|
Ok thanks for the explanation. Yeah to me running in snapshot or continuous is really an IP level thing, continuous being actually simply that the driver had an available buffer prior to the VSYNC, so I doubt this should be exposed to the application. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @KurtE, I did the review of the gc2145 part. I will do the review of the dcmi part also.
In a further update could you also split your commit ? gc2145 and dcmi are unrelated hence should be in separate commits. This help latter on when doing bisect or doing revert , fixes as well as backporting.
*/ | ||
typedef int (*video_api_set_snapshot_mode_t)(const struct device *dev, bool snapshot_mode); | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My understanding based on the conversation is that those modifications within video.h shouldn't be necessary, right ?
|
||
return api->set_snapshot_mode(dev, snapshot_mode); | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here as well.
/*{0x05, 0x01}, */ | ||
/*{0x06, 0x1C}, */ | ||
/*{0x07, 0x00}, */ | ||
/*{0x08, 0x32}, */ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Those are the Horizontal / Vertical blanking registers, could you explain the reason for changing them ?
.pixelformat = format, .width_min = width, .width_max = width, \ | ||
.height_min = height, .height_max = height, .width_step = 0, .height_step = 0, \ | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To avoid having twice the almost same macro what about having GC2145_VIDEO_FORMAT_CAP relying on GC2145_VIDEO_FORMAT_CAP_HL ?
Something like:
#define GC2145_VIDEO_FORMAT_CAP(width, height, format) GC2145_VIDEO_FORMAT_CAP_HL(width, width, height, height, format)
@@ -792,6 +809,9 @@ static const struct video_format_cap fmts[] = { | |||
GC2145_VIDEO_FORMAT_CAP(RESOLUTION_QVGA_W, RESOLUTION_QVGA_H, VIDEO_PIX_FMT_YUYV), | |||
GC2145_VIDEO_FORMAT_CAP(RESOLUTION_VGA_W, RESOLUTION_VGA_H, VIDEO_PIX_FMT_YUYV), | |||
GC2145_VIDEO_FORMAT_CAP(RESOLUTION_UXGA_W, RESOLUTION_UXGA_H, VIDEO_PIX_FMT_YUYV), | |||
/* Add catchall resolution */ | |||
GC2145_VIDEO_FORMAT_CAP_HL(128, 1600, 128, 1200, VIDEO_PIX_FMT_RGB565), | |||
GC2145_VIDEO_FORMAT_CAP_HL(128, 1600, 128, 1200, VIDEO_PIX_FMT_YUYV), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would be great to have macros for min width / min height, max width, max height instead of magic numbers here
@@ -1171,12 +1274,57 @@ static int gc2145_set_ctrl(const struct device *dev, uint32_t id) | |||
} | |||
} | |||
|
|||
static int gc2145_set_selection(const struct device *dev, struct video_selection *sel) | |||
{ | |||
LOG_DBG("called: (%p, %p: %u %u)", dev, sel, sel->type, sel->target); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not sure printing the dev and sel pointers really help here. Type are target values are fine.
return -EINVAL; | ||
} | ||
|
||
if (sel->target == VIDEO_SEL_TGT_CROP) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What about doing all the error / parameters checking first and then doing valid things are the end ?
if (sel->target != VIDEO_SEL_TGT_CROP) {
return -EINVAL;
}
return gc2145_set_crop(...)
|
||
static int gc2145_get_selection(const struct device *dev, struct video_selection *sel) | ||
{ | ||
LOG_DBG("called: (%p, %p: %u %u)", dev, sel, sel->type, sel->target); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same as for set_selection
struct gc2145_data *drv_data = dev->data; | ||
|
||
switch (sel->target) { | ||
case VIDEO_SEL_TGT_COMPOSE: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
get_selection and set_selection should be consistent I think. if there is a TGT_COMPOSE get then there should also be one for set. But it depend if you want to make this controllable by the application. Here compose would basically be to be able to manually control the ratio, but since it is already computed by the set_fmt, that is probably not needed.
.width = RESOLUTION_VGA_W, | ||
.height = RESOLUTION_VGA_H, | ||
.width = RESOLUTION_QVGA_W, | ||
.height = RESOLUTION_QVGA_H, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hum, when introducing the CSI support of the GC2145 I changed the default from QVGA to VGA since currently QVGA is not working in CSI. So moving it again would lead to breaking the CSI default now.
I need to check if everything is ok in CSI as well, CSI has some handling of internal FIFO that need to be computed and I am worrying that currently computation is ok for the VGA and UXGA but that's all ;(
Added support to video_stm32_dcmi for the new
set and get selection. These implementations simply forward the message to the underlying
camera object if they support these messages.
Also added support for a snapshot mode instead of
always using continuous capture mode. Tried
to make it semi-transparent when you desire it to be
The stm32_dcmi code now also allows you to work
with only one buffer. This will force it into snap shot mode. There is also new calls added to the api for: video_get_snapshot_mode and video_set_snapshot_mode.
That allows you to set it with more than one buffer and query what mode you are in.
GC2145 was updated first to try out these changes. The camera now allows me to follow the call order
that @josuah mentioned in another pr/issue.
With this driver I also updated it to allow more or less any video resolution:
{
.pixelformat = format, .width_min = width_l, .width_max = width_h,
.height_min = height_l, .height_max = height_h, .width_step = 0, .height_step = 0,
}
When resolution is set, it computes the scale factor. If you then later call set_crop, the same code is
used except it uses the ratios computed from the set_resolution.
With these changes: I was able to setup a test app, for the Arduino Nicla vision and send out a 480x320 image over USB.
More to come
Note: this is a replacement for #91975
My current test sketch/app is up at:
https://github.com/KurtE/zephyr_test_sketches/tree/master/camera_capture_to_usb
built using:
I am using the Arducam viewer with this on my PC. I am using the one at:
https://github.com/mjs513/Teensy_Camera/tree/main/extras/host_app
Picture using GC2145 on Arduino Nicla Vision shown on Arducam mini viewer.

Edit: current summary of changes:
There are several changes some of which will likely change if/when code reviews happen. Things like:
a) The stm_dcmi driver handles the get/set selection APIs and if the camera has also implemented these APIs, it
forwards the messages to them, else return error not implemented.
a1) GC2145 camera implements them.
b) Currently I allow arbitrary size of the frame GC2145, that is I have one fmt (per RGB...) which sets min and max versus the
ones where it current code which has 3 arbitrary sizes (1600x1200 - ratio 1, 640x480 ratio 2, 320x240 ratio 3). I instead
compute ratio and allow you to choose for example 800x600 which is less arbitrary than the 640x480... Note 320x240
computes ratio=5, except I currently limit to max ratio=3 per seeing other implementations that do so... Maybe should
make that max configurable.
c) Setting to allow the code to run in SNAPSHOT mode, which starts the camera, waits for one frame to come back and then deactivates.
This is the way that Arduino library works at least on MBED. Note snapshot mode also has some ability to recover from
failures...
d) Allow you to configure to only have one buffer, before it required at least two, If set to 1, it set forces SNAPSHOT mode.
With this running on Zephyr, I for example was able to program a Nicla Vision, which has no SDRAM and output at
480x320 over USB to an Arducam viewer. My test sketch (not sure what to call them on Zephyr)
is up at https://github.com/KurtE/zephyr_test_sketches/tree/master/camera_capture_to_usb
Also have others that output to Portenta H7 to an ST7796 display...