Add ability to use vector graphics for real time animation

[Bluebugs]

Checklist

• I have searched the issue tracker for open issues that relate to the same feature, before opening a new one.
• This issue only relates to a single feature. I will open new issues for any other features.

Is your feature request related to a problem?

Currently rendering vector graphics is done in software and uploaded to the GPU. This solution is slow and limit the ability to do real time animation. Having the ability to do real time animation open new use for fyne (animated interaction @60fps on most hw being the main goal).

I do not want to cover in this issue the problem that making an animated dashboard, a map, a dynamic graph, an interactive icon would require as those should be left to widgets of higher level that will use this new vector graphics primitive. This means that the problematic of generating an assemblage of triangle to get the shape you want from any source of data is left out of this proposal and considered the duty of higher level widget.

Also I believe it might be necessary to introduce masking operation of some sort to support all use cases, but this is also ignored on purpose by this proposal.

Is it possible to construct a solution with the existing API?

Existing API do not allow for building an efficient scene.

Prior discussion and work by @renlite on rounded rectangle would cover a small amount of the possible use case for a scene of vector graphics primitive, but I think that rounded rectangle are too useful and should exist as their own object and primitive.

Describe the solution you'd like to see.

Before discussing any further, I would highly recommend watching https://www.youtube.com/watch?v=QTybQ-5MlrE as it really show a very interesting path forward that is practical and doable which is the basis of this proposal.

As per the video, the base of the idea is that we should focus on triangle primitive to cover the use case we want. Following the above video, I guess we can start with just 3 types of triangle: plain, bezier curve and half circle.

I think as a first version we can avoid dealing with UV mapping and redirecting input. I think we should include the definition of stroke and fill color with a color being defined in a way that allow gradient, maybe just linear being supported.

The main problem with this proposal, is that this primitive are triangle while all current primitive are rectangular. This means that move and resize would likely have the same problem as on line and circle today and should be avoided as a design. Considering this, I would think the best solution is to introduce a new canvas object which is representing a scene/assemblage of the new triangle primitive. This would also allow the ability to make this scene follow the design of svg and define a view port in pseudo pixels unit using int instead of float which should avoid float precision issue when manipulating complex shape and large amount of triangle.

This new scene object can be easily resized and moved by the existing layout and code in fyne. Those operation would not affect the coordinate of the triangle inside and the rendering of those triangle can easily scale stroke and shape without approximation.

To summarize, we would have a new fyne.CanvasObject canvas.VectorScene which can render a new interface or type Triangle which has 3 points defining its 3 corners in integer value defined inside the canvas.VectorScene view port. There would be 3 differents renderer, one for half circle, one for full triangle and one for bezier curve. The half circle and bezier curve renderer will came with the ability to define a stroke (including color and width). The definition for the color has a lot of different possibility and it might be necessary to mark it as beta for a few release to allow for changing it if it doesn't match our need. I like linear gradient defined by a vector and that could be used for all the color definition (stroke, inside shape and outside shape). The half circle triangle might also have a starting and stopping angle to allow rendering corners, animated knobs or rotating spinner for example.

This would mean that we could rewrite the rendering logic of the Circle, Line, Rectangle and LinearGradient using this new canvas.VectorScene. I would still think it would be preferable to have the Rectangle one become a specialized rounded rectangle one instead, but it should be completely doable to generate a rounded rectangle using the primitive above.

The difficulty in this proposal is mostly, I think, into coming to agree on the API and figuring them out early in the next release cycle would be great.

[renlite]

https://github.com/memononen/nanovg as rendering fw?

• Antialized OpenGL backend, but min. version GL2.
• A lot of calculation is executed on the GPU in FragmentShader.

[Bluebugs]

nanovg is pretty cool, but I think we want to avoid adding any C dependencies that increase complexity on the developer when setting up their environment. Which is why I think we need to add this technology directly inside Fyne, but at the same time figure out the least complex way to do it.

[Bluebugs]

One thing I have not considered yet and I have no experience with is to use a tool to automatically translate C -> Go and maintain an independent fork of nanovg in Go. As I say, I have no experience with this kind of technology, but I would expect the resulting generated code to be not ideal and be difficult to maintain. On the other hand, this would give us a quick solution to support new rendering technology like Vulkan. It would likely be interested to more toolkit than just Fyne. I guess, the question is does anyone have experience with doing something like that? Is it a path we are interested into taking as part of the Fyne community?

[renlite]

I am thinking about your proposal and the idea to bring more shapes and the animation to the fragment shader (GPU) and to create complex shapes from three primitive vector objects (triangle, half circle, bezier curve) through composition in e.g. a widget.

Additionally I would like to mention that this considerations could include the render pipeline too, how some widgets or groups will be rendered on the screen. Will the GUI be rendered in one OpenGL canvas or would areas of screen use its own GL canvas. I think there are some experiences that more canvas would reduce the communication to GPU and the rendering if e.g. from 7 canvas on the screen only one has changed then only the changed areas in the render tree need to be handled.

Is there a documentation how Fyne's rendering works in detail? Does it make sense to think about the render process? IMO the topics belong together.

[Bluebugs]

The issue with multiple GL context is that it dramatically increase memory consumption. The benefit is that in theory you can do some rendering in parallel, but that hasn't been always reliable. It is a design that Vulkan is better at, but jumping a Vulkan painter seems a bit like a lot of work to me with no clear benefit when the current GL painter as a lot of potential improvement.

Currently every container end up being a buffer in which all its widget are rendered, as far as I understand. This means that we are using a lot of GPU texture as we nest container. It also means we are consuming a lot of memory bandwidth which impact performance on embedded devices. The benefit is that it make it easy to handle the Refresh call without the need to duplicate the state of the widget and make scrolling faster.

We need to address some race condition and that can't actually be done without duplicating the widgets state. Once that done, one of the benefit will disappear. The second one should be kept, but there should be a way to only buffer in the case of scrolling instead of always buffering.

Still, this change is not that high on the list as would be improving the text rendering first. Second will be grouping shader data upload. Every time we do send just a few triangle, we are building a communication with the GPU. The GPU is designed to get a lot of things in one go and process them. Sending small amount make it choke and wait on the next request. The solution to this problem is to package all the request for a same shader together and send them at once. Obviously you need to detect if the area covered doesn't intersect with other request that haven't been pushed and things like that. This reduce the amount of upload to the GPU, the number of shader change, use the streaming ability of the GPU better. Overall this will improve significantly our performance.

This would work well with pushing simple shape as described in this proposal as they would be gathered together and be pretty simple for the GPU to use. It should logically scale well.

[renlite]

I was able to group shader data upload to GPU for rectangle, round_rectangle. At the moment it should be possible to send following shapes in one request:

• rectangle
• round_rectangle
• circle
• horizontal and vertical line.

Because the shapes do not have a connection points it is not possible to use triangleStrip. The shape canvasObject is only a helper for testing to tell Fyne when to draw the shapes after rendering in a queue.
Next steps would be:

• Performance comparison for eg. 100 or 300 canvasObjects
• To include "Text" into the group_shader
• Find a logic when to draw grouped shapes.
• If it works as expected to refactor the code.

https://github.com/renlite/fyne/tree/develop
Example: https://github.com/renlite/fyne/blob/develop/cmd/example/flexrect.go

[Bluebugs]

Nice. Something that work for grouping is to have a stack of rectangular area that encompass the primitive drawn by each layer of the stack. When a new rendering operation needs to be done, it is dropped on the stack. It goes as further down as it can (without colliding with existing layer). Once it does, it rebound to find a layer that use the same shader and it get added to that layer. If no layer is found, a new layer is added on top of the stack. That stack is usually a static array (says 32 layers). When it is full, it is dumped on the GPU and reset.

[renlite]

@Bluebugs Thanks for you info.
It's now possible (milestone) to draw vector based objects (rect, circle, ...) together with textures (text) in one group_shader.
The shader is a prototype for the texture units, will change and be expanded to 32 texture_units in the future. This means it should be possible to draw all GL objects on the screen with one request if there are not more than 32 textures.
I think this could maybe possible instead of eg. 120 reqeusts for rendering 120 objects:
Example 1: 120 objects (20 textures included) = 1 Request
Example 2: 120 objects (40 texrures included) = 2 Requests (split by 33. texture)
I don't know the limits of a GPU for a vertex data stream. The data transformed to the GPU in form of attributes for an object is bigger than in a single upload.
Screenshot: A move over the Button creates only one upload to the GPU.

[Bluebugs]

This is an interesting approach. I must point out that there is a drawback especially on older hardware like an old Android phone where the GPU isn't great, complex shader are noticeably slower even for something that simple compared to what modern games do. We had benchmark this heavily in a past project and basically you want the shader to be as simple as possible and doing the selection on the CPU early was better (with the stack trick I described earlier).

[renlite]

With this approach I'm trying to improve the peformance in general and in particular of the webgl/wasm backend. The multi_shader should help to reduce the blocking communication between Js and Go and improve event_redraw disadvantage.
I think the group_shaders are not too complex. They are similar to the single_object shaders with only one extra "if, ifelse clause". In addition I moved the calculation of the normalization to the vert shader what should not be critical because it's called only for the vert_points.
The grouping is made on the lowest level of the painter, so the main rendering framework (logic) needs not to be adapted and the canvasObjects can be drawn in the same order as before. If an object is drawn over another, it should still work.

I don't understand the system of "layer of stacks" you described above.

• Does this drawing need a diffing that has to be done before the drawing level in the framework?
• Is this only possible for some areas of the screen that stay unchanged for the lifetime of the app?
• How to handle the correct order of painting for some canvasObjects? eg. Text over a Rectangle, or two Rectangles overlapping;

Would you please describe your approach further (picture)?

In theory using multi_shader should be possible without changing the rendering logic of Fyne and it should be possible to use single_shaders (Mobile) and multi_shader (Desktop, WebAssmbly(Browser)) in parallel.

[Bluebugs]

I am glad you are looking at making the wasm use case faster, thanks a lot to keep that one in mind. Still do not forget that it is working also on mobile browser :-)

Also in a shader every line is always executed. A if or for only touch a mask that will be passed to the instruction being executed inside the if to mask out which pixels to be computed by a specific line. In that regard, it is a complex shader as their is a lot of line that the GPU will have to go through.

Back to the idea of a stack. The principle is to figure out how to group rendering widget of the same primitive without having intersection problem. If you look at the UI, you have layers and a lot of them are not intersecting. Most of the time, the rectangle are all below and the text is all "above" from a visual perspective. So the idea is to build some form of stack to sort this layer out. To explain the algorithm, let's go through a few steps:

• First operation is a rectangle (A), it cover the entire window. It goes at the bottom of the stack as there is nothing in the stack.
• Second object is another rectangle (B), but it cover the half left of the screen. It can go on the same layer at the first rectangle (A) and be rendered after that previous rectangle (A).
• Third object is a text (C) on top of rectangle (B). It is also only on the half left of the screen. It can not go on the same layer as the rectangle (A, B) as it is a different shader. So it goes on top of the stack in a new layer.
• Fourth object is another rectangle (D), but it cover the half right of the screen. It does not cover the text (C), so it can join the layer with rectangle (A, B).
• Fifth is a text (E) over rectangle (D), and it only cover the half right. It can't go in the rectangle layer as it is a different shader, but it can be added to the text layer.
• Sixth is a thin rectangle (F) going left to right in the middle of the screen. It can't go below the text (C,E) layer as it collide with them. So it goes in its own new layer.

With this we have 3 layers in our stack (a layer as an associated rectangle used to encompass all the object it contain for faster collision handling):

• Top: rectangle (F)
• Middle: text (C, E)
• Bottom: rectangle (A, B, D)

Doing the rendering, we would do a bottom up operation, rendering A,B,D, then C,E and finally F.

[renlite]

Yes, I think WebAssembly will play a role in the browser for business apps and maybe even on the server. Fyne has mobile support, so wasm in mobile browser would propably not be the preffered way.

For better understanding I distinquish:
single_shader ... one call to GPU for every CanvasObject (current drawing)
group_shader ... one call to GPU for a group of the same type of CanvasObjects
multi_shader ... one call to GPU for all or a group of different types of CanvasObjects

Is it possible that a GPU has no jump- or goto-instructions for the if statement and it must walk through every line? The multi-shader (including rectangle, round_rectangle and texture at the moment) has twice LOC of the round_rectangle shader. It will be interesting to see the comparison between sindgle_shader calls and a multi_shader request on my old GPU (desktop).

Thanks for the explenation concirning group_shader of the same primitive type. To my understanding at the moment the painter (lowest level in the drawing process) gets the CanvasObject in the right order and has no information about the layout or the ui tree. The correct grouping of objects has to be done erlier on the call stack, somewhere on the walkthrough of the ui-tree. To avoid intersection the logic seems not to be trivial and the preparation of the groups for the painter can have an impact on the rendering performance. I haven't looked for the right place in the framework yet.

The advantage of a multi_shader would be that there is no complex logic for reordering the ui-tree necessary to avoid widget intersections. Eg. a cell of a table can overlap a text or a split-layout can leed to intersection problem.

In my POC I will try to implement the multi_shader and the group_shaders side by side, so maybe one technologie could be used or could offer a base coding for further work in the future.

[Bluebugs]

Indeed, GPU as they are vector machine are basically walking each instruction with a mask. Performance can be slightly improved by using uniform that allow the GPU to know that the value is going to be the same for all value in the vector and allow for jumping when that value is used in a if statement. This would still require to split call when we need to change the uniform to indicate we are using a different primitive. So back to square one.

I do not think we need to reorder call outside of the painter logic. We just need to push every request on a queue during the painter call and delay upload to once either the queue is full or the frame is done. Intersection can be made really straightforward by just building a rectangle that include all the primitive accumulated at each layer. It will produce a good enough result in practice.

I am looking forward to your progress on your PoC. I do not have much free time at the moment.

[renlite]

For the tests I have split the shaders into three directories on https://github.com/renlite/fyne/tree/develop/internal/painter/gl/shaders:

1. single_shaders
   Current drawing - for every CanvasObject one upload to the GPU
2. group_shaders
   For a group of CanvasObjects exists one shader. E.g. group_round_rectangle -> Rectangles, Circles, Horizontal-/Vertical Lines or a group_texture -> Text, Images. All CanvasObjects in a group will be send in one upload to the GPU. Textures have a cache limit of 32 objects GL and 16 objects GLES.
3. multi_shaders
   One upload for all CanvasObjects (vectors and textures). This is a theoretical test, because:
   a) If the cache limit of textures is full a new upload has to be created for drawing.
   b) I did not find the possibility to change the p.ctx.BlendFunc(...) in a shader. The text is not displayed in a good quality and in this case a grouping into vector shapes and textures(images) would be necessary.

For the group_shaders and the multi_shaders I needed to find a position in the framework to draw the batched data. For single_shaders the rendering and drawing happens at the same time for every CanvasObject but for the others the rendered vertex stream is collected to be drawn later in groups.
I don't know if the inserted code in the file https://github.com/renlite/fyne/blob/develop/internal/driver/common/canvas.go in method func (c *Canvas) walkTree(...) has the right position but it seems to work. Last line of the method: c.Painter().FinishDrawing()

Performance coparison between single_shaders and group_shaders

• Desktop:
  Without special tools I think the performance seems to be equal and if there is a difference I can't see it on my PC.
  For testing please use tags: go run -tags=group_shader /cmd/example/flexrect.go or go run -tags=single_shader /cmd/example/flexrect.go or other example file
• Web/Wasm:
  In the browser the group_shaders version on my PC is more responsive and I see a better performance.
  As far as I know fyne serve does't use tags. The default drawing is set to group_shader. To change to single_shader you have to change the build option in https://github.com/renlite/fyne/blob/develop/internal/painter/gl/draw_group.go //go:build !single_shader && !multi_shader and https://github.com/renlite/fyne/blob/develop/internal/painter/gl/draw_single.go //go:build single_shader.

On web there are events (hover/mouse in an area) that are handled much better and faster redrawing with group_shader is visible. Widgets like List and Table where the click-event is used for selection or scrolling too show better performance than single_shader but the performance is not satisfactory. Let's assume the "mouse over" and "click event" are submitted in the same way by the browser this could be an issue in some rendering part of the framework.

I suppose the Go-Wasm compiler does not support the experimental wasm GC at the moment and command fyne serve ... does not optimize compilation to wasm for production. Batched Group rendering seems to work better in WebAssembly where multithreading is still missing.

Layers logic is prepared in https://github.com/renlite/fyne/blob/develop/internal/painter/gl/draw_group.go. With batched rendering it should be possible to define areas (ID) that will never change or e.g. area A) that need not to be rerendered when an other area B) will change. In this case the rerender process, walk over the node tree, could be avoided and the cached vertex array for the area A) could be send to the GPU unchanged. Think of a master/detail-view, where the master area A) needs not to be rerendered when the user scrolls in the detail area B) (list or table).