display: add display_flush

[Finomnis]

Fixes #79798.

Indicates to the display that all previous writes should be flushed to the display.

This adds support for display drivers that internally contain a double-buffering or latching mechanism.
Currently the only way to implement those displays is to present all writes to the user immediately, which could cause tearing artifacts for multi-write renders. (it's quite common for UI managers to render to partial framebuffers in low-memory situations. LVGL, for example, recommends 1/10th of the screen size)

Open questions:

• Name of the 'end-of-frame' display driver api function?
  • display_flush (current PR code, conveys the intend pretty well that this is optional for displays to implement)
  • display_finalize_frame (might be a little too verbose)
  • display_present (maybe a little too strong, sounds like it is a must-use for displays)
  • display_flip (probably not because could be confused with x or y pixel order setting, like, 'flipping' the image)
  • display_pageflip (might be too usecase specific)
  • display_show (like present, might be too strong)

[pdgendt]

Introducing an API change should be accompanied with at least one driver that implements it.

That being said, this would tie the display driver closely to the lvgl way of working, which might require some more discussion.

[Finomnis]

Introducing an API change should be accompanied with at least one driver that implements it.

I don't think there is an upstream driver that requires this right now :/
An FPGA driver in our own project requires it.

That being said, this would tie the display driver closely to the lvgl way of working, which might require some more discussion.

I kind of disagree with this (that it ties to LVGL, i'm of course open to discussions). I think every display works in frames, and every frame has an end.

An alternative API would be display_start_frame and display_end_frame, but I don't really know a usecase where start_frame would be necessary. A display usually knows that a frame started when the first write arrives.

[Finomnis]

The problem I have is that regardless of LVGL, the display_write function nowhere states that it should be the only write per frame. And as it does not, there needs to be some way to signal double-buffered/latched displays that the modified content should now be displayed. This functionality is simply missing.

[Finomnis]

Introducing an API change should be accompanied with at least one driver that implements it.

Implemented it in dummy_display driver.

[Finomnis]

Is the name bothering? Would display_flush be better than display_finalize_frame? Or display_present, display_flip, display_show or similar?

EDIT: added a couple of options and thoughts in the PR description

[pdgendt]

This PR should be split into different commits:

• Introduce API function
• Implement in dummy driver
• Calling from LVGL

[pdgendt]

The problem I have is that regardless of LVGL, the display_write function nowhere states that it should be the only write per frame. And as it does not, there needs to be some way to signal double-buffered/latched displays that the modified content should now be displayed. This functionality is simply missing.

@danieldegrasse @jfischer-no can you weigh in here?

[pdgendt]

Introducing an API change should be accompanied with at least one driver that implements it.

This PR should be split into different commits

@pdgendt I feel like those are contradictory statements :) I'm happy to split it, as long as I don't have to merge them again because of the 'should be accompanied with at least one driver' requirement.

I am not contradicting myself: 1 PR, 3 commits please

[Finomnis]

Aaaah. Now it makes sense. Thanks for clarifying.

[jfischer-no]

Why are the commits and PR comments from different accounts/persons?

[Finomnis]

Why are the commits and PR comments from different accounts/persons?

Work email / private email. Is this a problem?
I can GPG sign them if requested.

I once tried (in a different PR) with Signed-off-by: Finomnis<finomnis@gmail.com>, but it didn't work for some reason.

[JarmouniA]

I once tried (in a different PR) with Signed-off-by: Finomnis<finomnis@gmail.com>, but it didn't work for some reason.

@Finomnis it should be Signed-off-by: Martin Stumpf <finomnis@gmail.com>

[Finomnis]

@Finomnis it should be Signed-off-by: Martin Stumpf <finomnis@gmail.com>

@JarmouniA Like this?

[danieldegrasse]

@danieldegrasse @jfischer-no can you weigh in here?

I'd like to better understand exactly what the case for a flush would be. Typically when using LVGL with a 1/10 buffer size, you need to write to the display graphics RAM when the buffer is presented, because the only RAM on the system actually capable of storing the full framebuffer is the graphics RAM on the display controller. In other cases (like the eLCDIF), the driver must define an internal framebuffer, and update it with the contents of the partial framebuffer each time LVGL sends a new one.

Typically the way display controllers deal with the possibility of tearing is via a TE (tearing effect) signal, which indicates when the display controller starts scanning a new frame from graphics ram to the LCD. Perhaps your FPGA could use something similar?

[Finomnis]

I'd like to better understand exactly what the case for a flush would be. Typically when using LVGL with a 1/10 buffer size, you need to write to the display graphics RAM when the buffer is presented, because the only RAM on the system actually capable of storing the full framebuffer is the graphics RAM on the display controller. In other cases (like the eLCDIF), the driver must define an internal framebuffer, and update it with the contents of the partial framebuffer each time LVGL sends a new one.

Typically the way display controllers deal with the possibility of tearing is via a TE (tearing effect) signal, which indicates when the display controller starts scanning a new frame from graphics ram to the LCD. Perhaps your FPGA could use something similar?

@danieldegrasse There's actually two sources of synchronization that could cause tearing:

• VSync; synchronization from the display to the frame buffer - If the 'swapping' of the content is not synchronized to the vertical sync of the scanline (I guess this is the tearing effect signal you are talking about), a random tear line can appear anywhere in the image
• Synchronization from the UI framework to the frame buffer - If we have multiple writes per frame, the information about which the last write is has to be propagated to the driver. Otherwise you could get tearing at the write chunk borders, if one write gets displayed during one frame and the next write (which is intended for the same frame) gets displayed at the next frame.

Even with a Tearing Effect/VSYNC signal, the chunk tearing would not be eliminated. The information about which write belongs to which frame, and at which write a new frame begins is simply missing from the display driver interface.

If LVGL renders in 1/10th screen chunks, all the driver sees is a continuous stream of write calls. So if it wants to use a VSYNC signal for tearing prevention, it needs to wait after the correct write call, which is currently not known to the driver.

To give you more detail about our concrete usecase, we have an FPGA with two frame buffers on it. The scanning speed of the FPGA is fixed (the UI is just an overlay to an FPGA-internal video stream), so we cannot sync the display to the UI, we need to sync the UI to the display. Further, our internal RAM is not enough to hold an entire frame, so we need to utilize LVGL partial renders.
We cannot utilize direct rendering, because for technical reasons this would be too slow. So we do need a partial frame buffer on the MCU.

Our procedure is:

1. Display scans from one FPGA buffer, while our display driver can write to the other FPGA buffer.
2. UI keeps sending write calls to our driver, which sends the data to the off-screen FPGA framebuffer
3. Once the UI is done with a frame, we request an FPGA frame buffer swap and wait for it to happen (like, wait for VSYNC)
4. Repeat, now rendering to the other buffer while the FPGA scans from the new frame.

For step 3, we need two sync sources. We need to know:

• which write is the last of the frame
• when the vsync happens

The vsync part could be done using a tearing effect signal which you mention, but the information which write is the last of the frame is simply missing from the display driver API, so we have no way to prevent the chunk tearing.

Hopefully that kind of explains the problem.

[danieldegrasse]

Yeah, this is a great explanation- thank you for laying it out. It seems like LVGL has lv_display_flush_is_last for this use case. Personally I see the reasoning for this API- I'm not sure how else we can do this effectively with partial rendering. Maybe we could add a flag to display_write? I think a new API is better here though, since we don't break downstream applications that way.

[Finomnis]

Maybe we could add a flag to display_write? I think a new API is better here though, since we don't break downstream applications that way.

Same thought. It would also be backwards compatible if we added it to the desc argument of display_write, but that feels like the wrong place.

[Finomnis]

@danieldegrasse there's also another reason for the extra api - the "flush" does not require the ui buffer any more. So it can happen after we give that buffer back to LVGL, and LVGL can continue rendering while we do the page flip and waiting for vsync.
(which is already how I implemented it, see my changes)

[faxe1008]

Thanks for the explanation @Finomnis. Change itself looks good. Can you maybe add a short sentence about this to the v4.0 release notes?

[danieldegrasse]

Considering this more, we need to clearly define when a user needs to call the display_flush API. I agree that there is potentially a need for this change, but we have an issue here because it fundamentally represents a different method of managing displays than we currently have in Zephyr today, right? Currently the user can assume that calling display_write will result in data being written to their display, but with this change that is no longer true.

If I understand the description of your downstream application right, no data would actually appear on the display until display_flush was called.

This changes display_write to effectively "queue" writes to the display, then display_flush passes those writes to the display.

One way I could see handling this would be to change the documentation around display_write to state that writes to the display may occur immediately or be queued, and a user must call display_flush to send all queued writes to the display.

I don't like this though, because this means all downstream applications need to start calling display_flush to use the API consistently. In some ways I would prefer if we added a parameter to the display_write API call that acts as a hint to the driver that the display should be flushed. This would break downstream applications, but it would do so in a way that was easy to fix. I could see users easily missing the new requirement to call display_flush, since it wouldn't be required for most displays, just required by the API definition.

@faxe1008 or @pdgendt do you have any thoughts here? I would also add that I agree if we make this API change there needs to be a full implementation of it. I think this could actually be done in the eLCDIF driver (or another driver using full framebuffers) using logic like so:

• if display_write is called, simply queue the write into a framebuffer
• when display_flush is called, pass that framebuffer to the eLCDIF hardware so it renders to the screen.

To be clear, I don't have an issue with the concept of changing the display API to deal with this issue- I just want to make sure we fully understand the implications of doing so, and communicate the change to downstream users effectively.

[faxe1008]

@danieldegrasse I agree that it is kind of ambigous if a write takes immediate effect or not. I like the extra API since it does not break downstream. Would it maybe be feasable to add a flag to the display capabilities screen_info field? The user still has to account for it but does not have to go hunting inside drivers sources to understand what the display wants.

[danieldegrasse]

@danieldegrasse I agree that it is kind of ambigous if a write takes immediate effect or not. I like the extra API since it does not break downstream. Would it maybe be feasable to add a flag to the display capabilities screen_info field? The user still has to account for it but does not have to go hunting inside drivers sources to understand what the display wants.

Yeah, I like this a lot. @Finomnis would you like some help getting this supported in a driver like the eLCDIF? We need to have a case of this being used in tree, and need to update the display sample (samples/drivers/display) as well to use it. I think adding a flag in screen_info makes sense, then the display driver sample can have additional handling with display_flush if that flag is set (same thing for LVGL).

[Finomnis]

I kind of thought the "double buffer" capability is exactly that, but it turns out to be something weird and obscure.

There's also a use of lv_display_flush_is_last in another eink display case already where blanking gets used in an (imo) incorrect way. This could also be removed after this change and instead be implemented through a flush.

Should we rework those? What's the point of the existing double buffer capability?

[faxe1008]

Docs state that "The display has two alternating RAM buffers", which imo is not necessarily the same thing. I could not find any intree usage of this flag. Adding a flag with an unambigous explanation would be good I think.

[jfischer-no]

I kind of thought the "double buffer" capability is exactly that, but it turns out to be something weird and obscure.

IIRC some EPD use double buffer to determine what area is to update.

There's also a use of lv_display_flush_is_last in another eink display case already where blanking gets used in an (imo) incorrect way. This could also be removed after this change and instead be implemented through a flush.

Yes, that can be used with some EPD displays that supports partial writes to display controller memory. Not sure why LVGL API is mixed here, this is just some graphics library.

Should we rework those? What's the point of the existing double buffer capability?

SCREEN_INFO_DOUBLE_BUFFER? I do not see it used in the tree.