[css-typed-om] use-case: data/audio visualization

[JohnWeisz]

Good evening,

Has data- and audio-visualization been considered as a possible use-case of the PaintWorklet? (both static and real-time audio/data visualization)

Currently, it seems there is no way to communicate with a PaintWorklet from other threads. If a MessagePort or a SharedArrayBuffer could be sent to the PaintWorklet, considerably more options would be available with this API:

let channel = new MessageChannel();
let buffer = new SharedArrayBuffer(1024);

await CSS.paintWorklet.addModule("vis.js");

CSS.paintWorklet.modules["vis.js"].postMessage({
  messagePort: channel.port1,
  buffer: buffer
}, [channel.port1]);

The other end of the port (or the buffer) could be sent to a worker/worklet thread.

Edit

On second thought, this is a little about being able to draw on a canvas from a worker thread directly, the outcome is very similar.

[tabatkins]

Currently, you communicate with a paint worklet by altering one of its arguments, or altering a property it depends on. Since you can pack whatever data you want into a custom property, this lets you pass arbitrary data over to it.

Have you tried this yet? We know this is probably insufficient for high-bandwidth communication with the worklet, and plan to address that by letting it receive an ArrayBuffer in some way, but haven't gotten there yet due to lack of demand. Please let us know if you have an example that requires that, where just altering a property is insufficient.

[JohnWeisz]

@tabatkins thanks for getting back with me,

"We know this is probably insufficient for high-bandwidth communication with the worklet, and plan to address that by letting it receive an ArrayBuffer in some way, but haven't gotten there yet due to lack of demand."

This is basically the core of my problem, it's not exactly efficient to transfer large amounts of continuously changing data, there's just so much overhead under-the-hood that it's not worth it.

What would it require to somehow get a MessagePort available here? And while we are at it, a MessagePort for AnimationWorklet as well.

[JohnWeisz]

Moreover, changing custom properties can only be done from the main thread, which makes it impossible to drive a PaintWorklet from a web worker or an AudioWorklet thread (without main thread interference, that is).

[stephenmcgruer]

@majido as an FYI for the AnimationWorklet MessagePort request

[majido]

We have thought about supporting posting message for Animation Worklet. See #869 for details here.

Unlike paint, Animation Worklet does not use custom properties so it does not have that channel. However we are not sure if post message has the right approach. See details on two issues here. In particular most of the time the information that needs to be communicated (e.g., viewport size changed) has relevance to a particular frame so being sync with that frame may be required. Maybe these issues are solvable though!

For Paint Worklet, I am not even sure how you would implement postMessage. Paint Worklet working model is completely stateless so introducing a side channel in form of postMessage would go against that e.g., you can no longer assume your paint only depends on input arguments and cache/resuse previous paint output. Also how useful is receiving a message when you cannot keep that message around beyond the immediate paint. What if we switch the global scope right after receiving the message?

[css-meeting-bot]

The Houdini Task Force just discussed cheaply passing binary data to worklets, and the implications on P&V and TypedOM, and agreed to the following:

• RESOLVED: Handle binary data by making a <blob> type in P&V, so a custom property can be set to a blob, and string-om can serialize it as `blob(...blob-url...)`; this utilizes the Transferable mechanism to handle snapshotting/invalidating. (Or possibly a slightly different KV store, will research this.)

The full IRC log of that discussion

<iank_> Topic: cheaply passing binary data to worklets, and the implications on P&V and TypedOM
<TabAtkins> github: https://github.com//issues/872
<iank_> TabAtkins: So several of the different houdini spec, folks when they use them, have usecases for passing large amounts of binary data to worklets. E.g. vertex data to a paint worklet, etc.
<iank_> TabAtkins: Its enough data that stringifying and passing through a custom prop is a bit of a performance issues, and we expect this to get worse over time.
<iank_> TabAtkins: We need a way to send typed arrays into worklets.
<iank_> TabAtkins: So we'll go through a few of the options.
<iank_> TabAtkins: First off - creating a new properties and values type which takes binary data, in the typedOM this would take an binary array, in the string APIs there probably isn't a reasonable serialization.
<giorgionatili> Giorgio Natili, Amazon
<iank_> TabAtkins: So that possibly works, there is a question around what happens with mutations.
<iank_> TabAtkins: E.g. do we need to snapshot it, then compare?
<iank_> TabAtkins: Feels straightforward from a design perspective, but lots of corner cases.
<iank_> TabAtkins: 2nd possibility, we hook up post message. This requires some new APIs, so they can be informed, or possibly it hooks up to a map, and we call you again when a new message has been issued. This sounds promising, b/c the contract looks simpler, postMessage has already transferrables, works on SABs etc.
<iank_> TabAtkins: Seems pretty promising, but we need to work on an API surface that would work.
<iank_> TabAtkins: Was there any other options?
<dbaron> I wonder how that interacts with worklet lifetime
<iank_> majidvp: You could do it on a per API basis, e.g. animation worklet has an options bag, which could be used.
<heycam> q+
<iank_> flackr: With paint worklet i
<iank_> flackr: ... i'm not sure this will work as there can be multiple instances.
<TabAtkins> s/be used/be updated to allow you to pass in more data later/
<iank_> flackr: Easier for the first option, however this first option doesn't work with transferrables.
<iank_> Rossen_: Are there any impl experience?
<iank_> majidvp: Audio worklet uses postmessage to do this.
<myles_> q+
<iank_> Rossen_: Whats the feedback so far.
<iank_> majidvp: This satifies the usecases.
<iank_> iank_: This works for audio worklet as they have one instance and they have a very strict processing model.
<astearns> anyone have a link for the audio worklet solution?
<rbyers> q?
<flackr> q+
<iank_> heycam: Not sure its a good idea, is it possible for blobs and blob uris?
<iank_> TabAtkins: Today the current model is base64 it.
<Rossen_> ack heycam
<iank_> iank_: TypedOM version of the base-uri?
<iank_> heycam: That's not what i was saying, but I think I perfer if we can get the postmessage solution to work.
<iank_> heycam: postmessage, what you really want is you call a function on the worklet to set a key-value into a map, and with postmessage its difficult to respond to it as occurs outside the other worklet functions.
<iank_> heycam: You could replay the messages as other instances spin up?
<iank_> dbaron: So one of my concerns is that - you can destory and re-create these worklets, and this doesn't seem compatible with the postmessage APIs.
<iank_> dbaron: You could do something where all the messages go into a store which persists
<iank_> rbyers: This sounds comparable to the typedOM.
<emilio> q+
<iank_> dbaron: When you shift between worklets you need to copy between the instances.
<rbyers> q+
<Rossen_> ack myles_
<iank_> myles_: I just wanted to bring up the fact the usecase the webgpu sends binary data to another process. They are going to do this with checkpointing. Would be good if these solutions were similar.
<iank_> TabAtkins: Does anyone have experience?
<myles_> s/are/might/
<TabAtkins> s/experience/experience with AudioWorklet, or should I go talk to the editors/
<iank_> majidvp: There is an instance and send message to that instance.
<Rossen_> ack flackr
<myles_> TabAtkins: Rafael Cintron at Microsoft
<iank_> flackr: On the first idea, around having something in the typed OM, if you had a const underlying array, you could pass a reference to this around, and not have the copy overhead.
<iank_> TabAtkins: That would be a new TypedOM API.
<iank_> flackr: A const typed array....
<TabAtkins> s/TypedOM/typed array/
<iank_> emilio: The issue with TypedOM is that you need to define how to serialize, e.g. calling the string APIs.
<iank_> TabAtkins: It might be ok that the string APIs just produce something like "[blob]"
<iank_> TabAtkins: referring to global named defining constructs inherited into the shadow dom.
<iank_> emilio: you could, instead of stashing the thing into typedom value - produce the name, have an extra function which is "blob(...)"
<iank_> TabAtkins: At what point are you describing a different API to the TypedOM get property API?
<iank_> TabAtkins: If you put blobs in a k-v store, ... how many k-v stores do you want to expose?
<iank_> emilio: This won't but the blob on the TypedOM?
<iank_> TabAtkins: Why not?
<iank_> emilio: But then you need to define how to serialize...
<iank_> emilio: Its basically the same issue as the props within the shadow apis.
<iank_> TabAtkins: I don't like breaking the invariant that you can access the value and re-set this?
<iank_> heycam: I feel like thats not a lot of work.
<iank_> TabAtkins: we could serialize to a base64 value....
<iank_> emilio: the issue is that you are creating many copies,... but not going to die on this hill.
<iank_> flackr: Could serialize to a blob-url....
<iank_> TabAtkins: A blob-uri is a short represenation to the array, which might work.
<iank_> TabAtkins: Register a blob-uri on the document global...
<iank_> emilio: blob-uris are global for the whole document.....
<iank_> TabAtkins: You have to be careful with name collisions, however blob-uris don't have that issue.
<iank_> flackr: One thing that you have to be careful, is that you need to know if they are valid or not, so that we can destory them when nessecery.
<emilio> s/blob-uris/registered custom properties/ on my comment above
<iank_> TabAtkins: hmm... yeah lifetime issues are interesting.
<Rossen_> q?
<Rossen_> ack emilio
<iank_> rbyers: Maybe this has been covered, the fundemental tension is that worklets are designed to be stateless... and people want to message state.
<majidvp> q+
<iank_> rbyers: Seems arch. preferable that this state flows through the TypedOM...
<iank_> rbyers: Anyone disagree?
<iank_> majidvp: One thing with ports is that you can pass them around. E.g. audio worklet plumbing into paint worklet to drive it.
<iank_> majidvp: But I don't know how strong of a usecase this is...
<iank_> TabAtkins: As long as you have a proper invalidation channel.
<iank_> iank_: But there can be many paint worklets.
<iank_> TabAtkins: We might want to look at transient binary data, but that doesn't stay around.
<iank_> flackr: The audio one usecase falls down, as could trigger repaint due to resize.
<iank_> heycam: With blobs you need to keep them around until you use the string....
<iank_> TabAtkins: I was talking before about shared-array-buffer... you still need the way for the audio worklet to drive the invalidation.
<iank_> flackr: But we need the SAB in the typedOM right?
<iank_> TabAtkins: yes...
<iank_> TabAtkins: That seems like the best way to do winamp viz.
<iank_> rbyers: How do you do synchronization?
<iank_> majidvp: Could use atomics?
<iank_> flackr: I think its up to the author to build the synchronization.
<bkardell_> q+
<iank_> (talking about painting while a SAB is chaning data while underneath).
<iank_> TabAtkins: For the main usecase the an ArrayBuffer value type seems like the right way and do with custom properties?
<iank_> rbyers: This has to be const... for the array buffer.
<iank_> <people seem to agree>
<iank_> TabAtkins: It might be worthwhile to do a separate K-V store, e.g. I depend on these keys vs. I depend on these properties.
<iank_> flackr: blob-uris have the contract with the typed array, but does require some one work to track references to them.
<Rossen_> q?
<Rossen_> ack rbyers
<iank_> rbyers: In most of these usecases i'm not sure you want to keep around older data.
<iank_> flackr: Thats true for winamp, however for lottie renderer you set the data once.
<iank_> TabAtkins: blob-uris are for blobs within your process, these are potentially not.
<iank_> flackr: Conceptually these are an immutable shared array buffer for these cases.
<iank_> bkardell_: This seems better for offscreen canvas.
<iank_> TabAtkins: Can audioworklet postmessage back?
<iank_> iank_: Yes.
<iank_> TabAtkins: Lets just talk about static binary data then.
<Rossen_> ack majidvp
<heycam> q+
<bkardell_> q-
<iank_> majidvp: A port is di-directional and can pass them around.
<Rossen_> ack heycam
<iank_> heycam: Would you ever want to animate these custom properties which are these arrays?
<iank_> TabAtkins: Everything can be animated, and then you do a binary flip.
<iank_> TabAtkins: If its a registered prop, you set a transition using this property... oh you couldn't do a transition/animation b/c it doesn't have a string apis,... oh but you can if we use blob-uris.
<iank_> heycam: Was just thinking about from the perspective from the animation if this works.
<flackr> iank_: the problem is the key is a string
<iank_> heycam: What is the lifecycle for the blob-uris?
<Rossen_> q?
<iank_> <discussion around revokedatauri and how this works..>
<iank_> flackr: TypedOM would always have a reference to the data even if revoke data uri was called.
<iank_> heycam: Are the string versions going to be the url() function?
<iank_> TabAtkins: no there will be blob() function as don't want to conflate with network access .
<iank_> heycam: On the other end, on the worklet api how do you read the data from the typed array?
<iank_> TabAtkins: The typedOM would expose the typed array directly.
<iank_> heycam: Oh i thought you'd have an extra step to get the blob-uri, then get the data.
<iank_> TabAtkins: Seems like an extra step for no good reason?
<iank_> bkardell_: blob blob blob blob
<Rossen_> s/bkardell_: blob blob blob blob//
<iank_> flackr: If the typed om took a url, but the value was only set after the url was fetched, you could plug in arbitary urls.
<iank_> bkardell_: There seems like there are peices within the platofrm
<iank_> TabAtkins: Has issues with things from the fetch api, e.g. can't fetch w/ credentials, etc.
<iank_> TabAtkins: Would prefer to keep this simple.
<iank_> bkardell_: There seems like there are peices within the platofrm
<iank_> TabAtkins: We should fix url however...
<iank_> flackr: Tab are you saying that we could upgrade to a full url eventually?
<iank_> TabAtkins: You could create a typedom way to create a credentialed request etc.
<iank_> TabAtkins: We plan on exposing custom hooks for fucntions to transform specified values into compuated at some point.
<iank_> TabAtkins: <summary of discussion> Future planned api should allow us to handle urls in custom props. without too much extra complication, and not changing our current discussion.
<majidvp> q+
<iank_> Rossen_: It sounds like we are done with this issue.
<iank_> TabAtkins: Lets get a resolution for this is how we are going to design it.
<iank_> majidvp: animation worklet is slightly different, as there is an options bag. but i think this proposal works nicely.
<iank_> majidvp: only thing that is missing is invalidating the options.
<flackr> iank_: this discussion is just about plumbing array data into the typed om and effects this has
<flackr> iank_: we can consider animationworklet separately like audioworklet did
<bkardell_> q+
<iank_> majidvp: If we consider options to be parallel to the stylemap to paint worklet... if you hook up an invalidation method, this would work. and we can add message passing.
<iank_> majidvp: Just wanted to point out the parallel, and it could work there as well.
<Rossen_> ack *
<iank_> bkardell_: I support the idea of being consistent as possible
<Rossen_> ack majidvp
<TabAtkins> proposed resolution: Handle binary data by making a <blob> type in P&V, so a custom property can be set to a blob, and string-om can serialize it as `blob(...blob-url...)`; this utilizes the Transferable mechanism to handle snapshotting/invalidating.
<bkardell_> q-
<iank_> bkardell_: Its a weak statement however.
<Rossen_> ack bkardell_
<iank_> heycam: Still have a suspicsion that a explicit K-V thing would be easier to work...
<iank_> flackr: I really don't thing we need a K-V store however, the key is the property, the value is the typed object.
<iank_> heycam: you are right that the custom props are a K-V, but they have additional requirements.
<TabAtkins> proposed resolution: Handle binary data by making a <blob> type in P&V, so a custom property can be set to a blob, and string-om can serialize it as `blob(...blob-url...)`; this utilizes the Transferable mechanism to handle snapshotting/invalidating. (Or possibly a slightly different KV store, will research this.)
<iank_> rbyers: Is this something we could try out with some customers?
<iank_> <folks - yes>
<iank_> Rossen_: Any objections?
<iank_> Rossen_: Resolved.
<iank_> Rossen_: Does that cover all three of these issues?
<iank_> Resolved: Handle binary data by making a <blob> type in P&V, so a custom property can be set to a blob, and string-om can serialize it as `blob(...blob-url...)`; this utilizes the Transferable mechanism to handle snapshotting/invalidating. (Or possibly a slightly different KV store, will research this.)
<iank_> RESOLVED: Handle binary data by making a <blob> type in P&V, so a custom property can be set to a blob, and string-om can serialize it as `blob(...blob-url...)`; this utilizes the Transferable mechanism to handle snapshotting/invalidating. (Or possibly a slightly different KV store, will research this.)
<iank_> TabAtkins: The post message issue was about paint worklet. and agreed that right now use offscreen canvas, and if we need it later can consider.

[guest271314]

@majido

Also how useful is receiving a message when you cannot keep that message around beyond the immediate paint.

If gather this proposal correctly the messages might not need to be kept.

The previous/current value can be kept in the HTML element at the CSS value at the element where paint() is defined.

E.g., given initial CSS

--data:'["initial data"]';

As @tabatkins mentioned, arbitrary values can be set at paint() function or at properties

  await CSS.paintWorklet.addModule("vis.js");
  
  const paint = document.querySelector("p");
  
  const paintWorkletMessage = ({element = p, prop = '--data', message = '[]'} = {}) => {
    element.style.setProperty(prop, message);
  }

  await new Promise(resolve => setTimeout(resolve, 200));
  
  paintWorkletMessage({message: `["other arbitrary data"]`});
  
  await new Promise(r => setTimeout(r, 200));

  paintWorkletMessage({message: `["more arbitrary data"]`});
  
  paint.onfocus = e => {
    paintWorkletMessage({message: JSON.stringify([...Array(Math.floor(Math.random() * 10))].map(_ => Math.floor(Math.random() * 2)))});
  }

at the paint worklet

static get inputProperties() { return ["--data"] }
// ..
paint (ctx, geom, properties, args) {
  const data = properties.get('--data').toString();
  const message = JSON.parse(data);
  console.log(message);
}

If #857 includes providing a means to set text at --data property from within the paint worklet the updated set value can be read outside of paint worklet scope.

What if we switch the global scope right after receiving the message?

The scope can be checked for properties defined within paint worklet scope, which allow for the same code to be used for different scopes

if (!("registerPaint" in globalThis)) {
  globalThis.registerPaint = function(...rest) { }
 console.log(globalThis);

}

[Jamesernator]

On second thought, this is a little about being able to draw on a canvas from a worker thread directly, the outcome is very similar.

What you're probably looking for is OffscreenCanvas, this is basically a canvas that can be posted to a worker. Unfortunately the support for it is poor (only Chrome).

If you can generate visualizations pixel-by-pixel rgb data then ImageData is probably what you want as it can be efficiently post-messaged from the worker. It's what I use due to the lack of support for OffscreenCanvas and have generally found it's performance fine.