Image capture delivers empty result and finally goes to endless loop

[Spontifixus]

Hi,

first of all thank you for this great library! I have encountered a problem actually - and it might well be that I have misunderstood something here or used the library in a wrong way!

The scenario:
I try to capture an image every 5 seconds and store this to an Azure Blob storage. In a first step I went with the time lapse feature, but ditched that, because I could not find a way to resize the image before sending it to the capture handler. For this approach I created a dedicated AzureStorageCaptureHandler that was based on the MemoryStreamCaptureHandlerwhile implementing the IFileStreamCaptureHandler interface.

So I went and implemented my own mechanism. You can find a very stripped down implementation in the Spontifixus/mmalsharp-demo repository. The setup is derived from the "resizer component" example. In pseudo code it looks this way:

ConfigureCamera
For i = 0 To 100
    memoryStreamHandler = new MemoryStreamCaptureHandler()
    CaptureImage(memoryStreamHandler)
    StoreImage(memoryStreamHandler)
    memoryStreamHandler.Dispose()
Next

Expected behavior:
I expect the code to store 100 images, without capturing empty images, and without going into an endless loop.

Actual behavior
The application produces empty images every few captures and finally goes into an endless loop because a buffer cannot be released. Here's a log of the test run: debug.log, with the following interesting parts:

• Lines 105 - 154: Successful capture
• Lines 155 - 162: Successful storage
• Lines 495 - 542: Failed capture
• Lines 3051 - End: Capture Started, resulting in infinite loop.

[Spontifixus]

I have had a short look at the sources - and a reason for this endless loop could be the call to the Dispose-method at the end of the MMALBufferImpl.Release()-method. I could imagine that during the disposal somehow a release is triggered once again. I don't have the means to debug this on my Raspi unfortunately...

[techyian]

Hi,

Sorry in the delay getting back to you. I initially had a look into this using Mono and wasn't able to reproduce it. I then moved onto .NET Core 3.1 and immediately I could reproduce the issue. It seems quite intermittent though even on .NET Core 3.1. I suspect it's going to be a difficult one to figure out.

I was wondering whether you could please set MMALCameraConfig.Debug = true; at the beginning of your application and see if you can reproduce it - the verbose logs may help track down what's happening here as they include buffer metadata. If you struggle to reproduce with that set, it may be a timing issue and I may need to look into re-adding the thread locks in the native callback methods.

[Spontifixus]

With the debug output enabled reproducing the issue was a lot harder than without. I had to run the sample app several times, but finally it "worked". Please see the debug.log for the failing image capture.

[techyian]

I've just committed a change which I hope will fix this. The change prevents a buffer being sent to an output port if we have reached the end of stream. I've not been able to reproduce the issue again with this change in place, could you have a test and see if it fixes it for you please? You will need to clone the dev branch.

[Spontifixus]

That seems to fix the defect, indeed. Yet occasionally (though by far not as often as before) I still see failed captures (i.e. the resulting stream being empty).

[techyian]

Ok well it seems like we're making progress here so that's good. With regards to the stream being empty, have you observed the logs whilst the verbose logging is enabled to see if the buffer header has any metadata to suggest why the stream may be empty? Are you able to reproduce the stream being empty with your example application you linked to?

[Spontifixus]

Hi, thanks again for your support! Here's a log file with at least one failed capture. You can find the relevant log entries at lines 3535ff: debug.log

There's a difference between successful and failed captures. Here's a successful capture:

   Buffer Header 
  ---------------- 
  Length: 59082 
  Presentation Timestamp: 299252705 
  Flags: 
  MMAL_BUFFER_HEADER_FLAG_FRAME_END 
  ---------------- 

And this is the header of the failed capture:

   Buffer Header 
  ---------------- 
  Length: 0 
  Presentation Timestamp: 0 
  Flags: 
  MMAL_BUFFER_HEADER_FLAG_EOS 
  MMAL_BUFFER_HEADER_FLAG_COMPLETEFRAME 
  ---------------- 

I can see that the length is zero and the flags signal something different - but I lack the knowledge about both the hardware interface and the JPG-encoding to make any sense of it...

[techyian]

That's certainly a bit bizarre. If you're using the ImageEncoderComponent connected to the camera's still port then MMAL_BUFFER_HEADER_FLAG_EOS should not be seen in the buffer's metadata, MMAL_BUFFER_HEADER_FLAG_FRAME_END signifies EOS in this instance. Can you confirm that you can reproduce this behaviour with the example application you've got here? Unless you're doing anything strange with your setup I doubt it's this library causing that behaviour, especially considering the image taken after is successful.

[Spontifixus]

The behavior was reproduced using that exact code - except that I modified the logging a bit to be able to find the failed capture more easily.

Usually this behavior poses not much of a problem, due to the fact that I capture the next image immediately if one capture went wrong.

In the real world this code does not run in a for-loop though but rather in a while(isCaptureActive)-loop, and that stopped working after a few hundred captured images. I don't have detailed information as of now, but I will update this issue should I come to the conclusion that the reason is at MMALSharp or maybe on even lower parts of the system.