PIXFORMAT_RAW support missing? I want to generate DNGs....

[aggregate]

Although the JPEG and GRAYSCALE formats are fairly useful with the OV2640, I'm trying to get maximum RGB image quality and I want to do my own Bayer conversion. The catch is, there doesn't seem to be any support for PIXFORMAT_RAW in the driver, let alone any way to tell it I'd like the 10bpp rather 8bpp version of that. Basically, my plan is to wrap the truly raw sensor data as an uncompressed DNG and do the Bayer conversion later; I'm even happy to do my own bad-pixel repairs. Dave Coffin (the author of DCRaw) has a simplified DNG-wrapping program elphel_dng.c, but it's even easier to just stick a pre-computed DNG header on the data. Anyway, I'd be perfectly happy with the driver just allowing me to get the 8 or 10bpp raw sensor data without a header. Is that doable?

[me-no-dev]

the problem is that this would be too much data for the ESP32 to be able to acquire and process. We can try this with our newer chips (S2/S3) soon. Most boards are not wired to support the 10bit data bus also, so 8bit would be the max

[Schaggo]

I would be very open to contribute to an open ESP-pro cam project.
I don't know what of the following would be achievable, but a system with the following capabilities would be incredibly useful and could potentially compete with more expensive systems:

-Multi-device frame synchronization
-RAW output
-Ethernet

Last year I built a cam array with 72 ESP-cams for photogrammetric 3D-Scanning, which is surprisingly capable but still rather a hobby than a product. With the above-mentioned capabilities, people could potentially build their own cinema-grade scanner for less than 1000€ -depending on the module price of cause.

[me-no-dev]

We do not have enough pins for Ethernet. I need to check if ESP32-S3 will have Ethernet, because it was removed from S2.
Frame synchronization... hmmm... not an easy task :)

[aggregate]

the problem is that this would be too much data for the ESP32 to be able to acquire and process. We can try this with our newer chips (S2/S3) soon. Most boards are not wired to support the 10bit data bus also, so 8bit would be the max

8bpp raw should be the same data size as gray. I do see timeouts when trying to capture full resolution gray... is that the same problem? I would have expected 1 buffer of raw 1632x1232 to fit in the PSRAM, although the JPEGs are quite a bit smaller. There is still the possibility to use the ROI support to send portions of a raw frame at a time, although then the portions would be from different captures.... I suppose without 10bpp, the JPEGs might actually give better data than raw in terms of dynamic range (assuming all 10bpp are used in the OV2640 JPEG pipeline).

Incidentally, I'm a computer engineering professor at the University of Kentucky. One of the things I'd be using this for is Lafodis160, a 160mm-diameter LArge-FOrmat DIgital Scanning camera that I published on at Electronic Imaging 2021 in January. Currently, it is sampling each position using JPEGs. The saving grace for it is that I'm normally only trying to get the equivalent of 5um pixels for the stitched capture, so multiple 2.2um sensor pixels contribute to each spot and the JPEGs don't hurt image quality too much.

[aggregate]

I would be very open to contribute to an open ESP-pro cam project.

Obviously, me too. I've been using CHDK with Canon PowerShots for over a decade... for example, I'm the guy who wrote the Lua reference card for CHDK.

I don't know what of the following would be achievable, but a system with the following capabilities would be incredibly useful and could potentially compete with more expensive systems:

-Multi-device frame synchronization
-RAW output
-Ethernet

Last year I built a cam array with 72 ESP-cams for photogrammetric 3D-Scanning, which is surprisingly capable but still rather a hobby than a product. With the above-mentioned capabilities, people could potentially build their own cinema-grade scanner for less than 1000€ -depending on the module price of cause.

I actually have an undergraduate senior project team in the early stages of working on an "ESP32-CAM Swarm" project trying to do single-point control and precise wireless sync of a large number of ESP32-CAM-based camera modules. How did you do your sync? The obvious thing would be using UDP broadcast, which they're supposed to be testing, but they're also looking at other methods, such as using NIR to sync. They also haven't yet done testing to determine precise lags and variability in triggering the OV2640. If you have pointers to any of your work WRT ESP32-CAM sync, that would be appreciated....

[Schaggo]

Hey, feel free to check out my ESP 3D Scanner project - I will update my horrible code within the next 10 days or so.
Small disclaimer: I barely know what I am doing. I am a solo game dev and a really bad programmer that wants to see how close to AAA asset quality I can get with a limited budget - this is the hole motivation.
The project is kind of on hold, because I just built a scanner using 12 Raspi HQ cams, but I will return to it later this year.
The project does not use the latest driver and is a patchwork of weird functionality (S3 firmware updates, QR-code to configure,...)

You mentioned one RAW grey not fitting in the PSRAM. I found out that anything crossing over 50% of the total PSRAM size gets corrupted. Limiting the PSRAM to half its size kind of mitigates this for JPG at quality 0 - meaning I am able to get up to almost uncompressed JPG up to 955kb - RAW grey does not cause issues. This "hack" is not yet in my repo, but was easy to implement.

Synch is a rather relative word, because of the mentioned variability regarding the OV2640 (or OV3660).
But with multicast UDP and NTP from the WIFI router I am able to reliably achieve to trigger an LED blink with a total delay from first to last of roughly 12ms over all 72 cameras (measured via 240fps video).

Bottlenecks of my system are my abysmal code and the way my WIFI router handles UDP and limit of max simultaneous connections.

If I were your student, I'd try with 433MHz - periodically sending a times stamp (now [0]), and control via command and generously spaced execution time stamp, but I'm also an idiot...

I am trying to check out your Lafodis project, but the pdf is not loading ...

[Schaggo]

The PDF loaded ... I am stunned!

[aggregate]

Hey, feel free to check out my ESP 3D Scanner project - I will update my horrible code within the next 10 days or so.
Small disclaimer: I barely know what I am doing. I am a solo game dev and a really bad programmer that wants to see how close to AAA asset quality I can get with a limited budget - this is the hole motivation.

Looks good at a quick glance. I'll point my students at it. :-)

You mentioned one RAW grey not fitting in the PSRAM. I found out that anything crossing over 50% of the total PSRAM size gets corrupted. Limiting the PSRAM to half its size kind of mitigates this for JPG at quality 0 - meaning I am able to get up to almost uncompressed JPG up to 955kb - RAW grey does not cause issues. This "hack" is not yet in my repo, but was easy to implement.

Very interesting. If PIXFORMAT_GRAYSCALE works, 8bpp PIXFORMAT_RAW should be doable too.

Synch is a rather relative word, because of the mentioned variability regarding the OV2640 (or OV3660).
But with multicast UDP and NTP from the WIFI router I am able to reliably achieve to trigger an LED blink with a total delay from first to last of roughly 12ms over all 72 cameras (measured via 240fps video).

12ms is pretty good; I doubt we can get the OV2640 to trigger much more consistently than that.... The idea of using sync to establish global time and then triggering at agreed-upon times is an old standard for us. For example, we advocated that in
Lessons from design, construction, and use of various multicameras. It was my group, back in Feb 1994, that built the world's first Linux PC cluster supercomputer... which had hardware barrier synchronization running through the parallel printer ports... broadcast took about 3us and we had a protocol where we could get global agreement on time to within about 0.5us.

I am trying to check out your Lafodis project, but the pdf is not loading ...

Sorry about that. For political reasons, I don't host Aggregate.Org (my research group website) on nodes in my supercomputing machine room... instead, it's on a University of Kentucky virtual host that keeps throttling it because it gets "too much" traffic. It will eventually load if you keep hitting it. IS&T should have the paper freely online soon, but it usually takes a few months after the conference before papers are posted.

[Schaggo]

About the cam module: do you know if the variability in capture time comes from the driver or from the cam module itself?

[aggregate]

About the cam module: do you know if the variability in capture time comes from the driver or from the cam module itself?

Not yet. It does seem like the OV2640 is capable of very precise external sync, but I'm not even sure the wiring allows it here.

[Schaggo]

Thanks! Will investigate.

[jameszah]

I found out that anything crossing over 50% of the total PSRAM size gets corrupted.

Is that "literal"?
You get an error in any block crossing the 50% boundary. So you can use the low-side and the high-side but avoid the gap?

[Schaggo]

Yes. Images taken with full psram often get corrupted which leads to incomplete images. Limiting the psram solves this. Don’t know if this still applies to the latest software, but it works for me.

[jameszah]

Wow - I never considered that. I've been unraveling the jpeg to look for strange elements in the huffman codes, to find bad jpegs.

So the magic number is 3FA0 0000.
I was allocating buffers for the biggest jpeg on the ov5640, with 3 buffers of 983056 bytes, so the addresses were:
3F80 0044, 3F8F 0054, 3F9E 0064 ... and the 3F9E 0064 + F0010 (983056) crosses the magic number.

I'll try avoiding that buffer!

[Frudawski]

I had the same requirements and looked a little into the code a few month ago. Unfortunately it seems, that 8bit RGB or RAW is not possible with the current version of the ESP32. The driver has no raw support implemented and RGB888 is basicly RGB565 upscaled to 256 as max value. So you will only get 16 bit colour depth. The JPEG format might have some advantage here.

I did try to solve that issue and experimented a little with the code. I changed the registers to RGB888 disabled all sensor features and changed the DMA mode but it seems, that the ESP32's ISP is not fast enough to catch the data from the sensor in time. I ended up with missing pixels and was unable to resolve this issue.

I am also open to contribute to that matter but I am not an expert. I am a PhD student at the technical University in Berlin and I am using the esp-cam modul as a portable light measurement device, or so is the plan.

[github-actions]

This issue appears to be stale. Please close it if its no longer valid.