What is preventing Event Hub from working in Node?

[ffMathy]

Currently, according to the documentation, event hub doesn't work with Node.

From http://ftrack-javascript-api.rtd.ftrack.com/en/stable/handling_events.html:

The event hub in the JavaScript API has some minor differences and lacks some of the features available in the python counterpart.

However, from my point of view, it looks like the socket.io client should work in Node out of the box: https://stackoverflow.com/questions/10703513/node-js-client-for-a-socket-io-server

Can you elaborate on what exactly isn't working for Node?

Because in that case, we could be interested in fixing it and contributing a pull request.

[gismya]

Since updating versions of socket.io would incur breaking changes to our API, affecting customers' API clients, we are still on a version of socket.io that to my knowledge does not have a Node-compatible implementation.

[ffMathy]

Aha. Interesting. Could this be part of the 2.0 milestone as well then?

[gismya]

The main issue isn't with the client but with the server portion of the code. I know there have been discussions on updating that, but it's to my knowledge not on any current roadmap. I'll forward the request to the relevant team, always useful to have specific needs to fulfill when prioritizing.

[ffMathy]

Can you elaborate what you mean by "server portion"? Is this the portion that allows the event hub to publish events?

I was under the impression that the server portion would be in Ftrack itself, and that the NPM package would just be a client.

[gismya]

That is exactly what I mean, the issue is that the socket.io implementation in the backend ftrack code is for an older version of socket.io.

[ffMathy]

Aha. I see. Which version of socket.io is the server using? Do we know? Maybe I can try to figure out if it would be breaking or not, or research in what it would take to add support to the existing one.

[gismya]

The versioning for socket.io seem to match the client and the server, so should be 0.9. The 0.9 API seems to be close enough to the 1.x API that it'd probably check if the last 1.x release supports Node and check if that would work in this API client.

[ffMathy]

I have confirmed that there is no mention of Node support in 0.9.17:
https://github.com/socketio/socket.io-client/tree/81089a2723770d5b32753485d1e5a160a72abab6

And that there is an explicit mention in 1.0.0:
https://github.com/socketio/socket.io-client/tree/367a9cb5a51d146e7e4bb1dd0d340eaf80de29ce

I will try running some experiments, and then I'll make a PR.

[ffMathy]

I created a pull request now over at #96. It's a draft. It doesn't seem to work yet, but I am unsure if it is just because of some corporate VPN we have at work or something.

But perhaps it could be a starting point for you to go further, since I think I am a bit stuck? Right now, I am experiencing a timeout error. But the code compiles and runs.

Here is the code I used to experiment:

import {Session} from '@ftrack/api';

const client = new Session(
    process.env.FTRACK_SERVER as any,
    process.env.FTRACK_API_USER as any,
    process.env.FTRACK_API_KEY as any
);

//this works.
client.query("select id from Project").then(console.log);

//this gets connection timeout.
client.eventHub.connect();

client.eventHub.subscribe('topic=*', (x: any) => {
    //this is never called due to the above timeout.
    console.log(x);
})

If you want a better repro example, I can also create that.

[ffMathy]

If this could be fixed, we would be forever grateful. I suspect my branch might be connecting forever because of VPN or company firewall. Would be awesome if someone could try my branch on their machine, or perhaps solve the problem.

[gismya]

I'll see if I can look into this! We also saw and started discussing the pull request internally last week. Sorry if it feels like you were shouting into the void for a few days. The easter holidays made the internal resources a bit tied up.

[ffMathy]

Oh, not at all! And I'm super happy you're looking into this.

Right now, we have a server which forwards all of our events to an AWS SNS topic.

Our plan is to then have AWS Lambdas (serverless functions) that respond to these. But for them to be able to "reply" to events like action.discover and action.launch, they need to be able to use the event hub.

And we do not have a lot of Python competencies in house, and Denmark is not very good for finding Python competencies in general, compared to other languages.

So until this somehow works, we're kind of stuck.

[ffMathy]

I now actually have time for this, but outside of working hours. And then I am not able to access my work computer, which has the company VPN, which can reach the Ftrack instance.

That's problematic. It would be super awesome if Ftrack offered some dummy cluster that everyone could use when contributing, or perhaps request access to.

[ffMathy]

I decided to go pick up my work computer and made another "interesting" experiment PR here:
#98

In this one, I tried "sacrificing" web compatibility temporarily, just to see if I can hack the existing socket.io-client code directly to work in Node.

I switched from using XMLHttpRequest to using https://www.npmjs.com/package/xmlhttprequest which also seems to be used by version 1.0.0.

Now, the "connect()" call is no longer freezing forever, so that's very interesting actually. Unfortunately it's timing out after a while.

I feel like we're getting closer though! I am inserting debugger; statements everywhere and attaching a debugger in Node to see if I can find the root cause.

Wireshark etc doesn't seem to be installable on my work machine (some policy thing I think).

[ffMathy]

@gismya can you tell me more about the Event Hub API on the server-side? Is it just 100% websocket based? I don't know much about websockets. Can they be simulated by regular HTTP calls, or do they use TCP?