Live Cursors Interpolation

[suvamsh]

We're using the ably live cursors (via spaces) for a collaboration feature in our product. When clients are using different sized screens the cursors are not being interpolated correctly hence they show up in the wrong position on clients. I've attempted to solve this by converting the (x,y) position from pixels to a percentage of the canvas but it is still wrong for the horizontal X axis.

percentage = (position / canvasLen) * 100

Thew above formula is applied to the X and Y axis. I've also tried to scale the pixel positions like below (which also doesn't work):

    function transformCoordinates(
        client1_x: number,
        client1_y: number,
        client1_width: number,
        client1_height: number,
        client2_width: number,
        client2_height: number
    ): [number, number] {
        // Calculate the scaling factors for width and height dynamically based on the x-coordinate
        const scale_x: number = client2_width / client1_width;
        const scale_y: number = client2_height / client1_height;

        // Adjust the scaling factor based on the x-coordinate (you can customize this adjustment as needed)
        const adjusted_scale_x: number = scale_x * (client1_x / client1_width);

        // Apply the adjusted scaling factor to the x-coordinate
        const client2_x: number = Math.floor(client1_x * adjusted_scale_x);
        const client2_y: number = Math.floor(client1_y * scale_y);

        return [client2_x, client2_y];
    }

Please share any examples or cases of how to do this using Ably.

[dpiatek]

hey @suvamsh - I've created a repo with an example implementation, https://github.com/ably-labs/responsive-cursors-example.

It also uses scaling and shows the potential problems of fixed/dynamic elements within a canvas.

It's a bit tricky to get a sense of what might be wrong just from the sample you posted. adjusted_scale_x - could you clarify why that is needed, and specifically for x coordinate and not y?

[Keegs12]

hey @dpiatek - Just to give some clarity on the adjusted_scale_x, that was simply an attempt on our part as the x positioning appears to be the problem. Our y positioning is just fine, yet our x is off, so we attempted with an adjusted_scale_x to see what type of effect it had, it really didnt do much haha.

Anyways, I've since attempted using the example implementation you provided above, running into the same issue, its just simply not lining up, again mainly with the x positioning being the problem.

Below I'll list some snippets maybe you can look at to see if I missed something, but the math between our attempts (not necessarily the code that was provided) is extremely similar to yours.

To also give more clarity we are not using the full width of the screen, our canvas take up a rough % so we use the width and height of it, instead of the window

Setting the position and canvas values inside the emitted message
const bounds = liveCursorsDiv?.getBoundingClientRect();
            if (!bounds) return;
            let relativeLeftPosition = e.clientX - bounds.left;
            let relativeTopPosition = e.clientY - bounds.top;
 
set({
        position: { x: relativeLeftPosition, y: relativeTopPosition },
        data: {
             state: "move",
            canvasWidth: liveCursorsDiv?.clientWidth,
           canvasHeight: liveCursorsDiv?.clientHeight,
      },
 });

Calculating our scale and interpolated position
const scaleX = viewerInstanceRef?.canvas?.clientWidth! / cursorUpdate?.data?.canvasWidth!;
const scaleY = viewerInstanceRef?.canvas?.clientHeight! / cursorUpdate?.data?.canvasHeight!;
const x = cursorUpdate.position.x * scaleX;
const y = cursorUpdate.position.y * scaleY;

then x and y are our left and top positions

Not sure if maybe I messed something up, but this was based off your github repo