"unexpected command received" message on backend

[justinclift]

These two exception messages showed up in the production API server logs a few hours ago, though nothing seems to be actually wrong with the backend:

Apr 25 09:21:29 api.dbhub.io api[364572]: 2023/04/25 09:21:29 Exception (503) Reason: "unexpected command received"
Apr 25 09:21:29 api.dbhub.io api[364572]: 2023/04/25 09:21:29 Exception (503) Reason: "unexpected command received"

That unexpected command received isn't a string in our source code, but instead seems to be coming from the AMQP library.

Doing some searching online shows a few other instances of the message, and it seems to mean our AMQP queuing code isn't quite right:

• Get "unexpected command received" error when reusing a channel streadway/amqp#170

I'll take a look at it properly after the current Stripe task, as the explanation given in that issue doesn't seem to line up with the RabbitMQ documentation. Probably just need to understand something better, at which point it'll make sense. 😉

[justinclift]

This Hacker News post from a few weeks ago is likely relevant, as they seem to have experienced the same problem with their RabbitMQ usage (even using the same AMQP library as us):

    https://news.ycombinator.com/item?id=35526846

This particular comment likely describes the root cause. And the root problem seems to be something we're doing too, as we're ack-ing messages at the end of each database operation.

[justinclift]

We might be able to get away with something like this in the meantime instead, dropping the need for RabbitMQ:

https://github.com/rverton/pgjobs

I'll take a look at this stuff later on.

Hmmm, probably not. This seems to use a polling based approach, which isn't compatible with our needs. Our users expect sub-second responses to their buttons clicks for live databases, and I'm pretty sure polling our backend database 10+ times a second isn't really the right approach.

---

Actually, we could probably drop the polling behaviour and instead use LISTEN/NOTIFY to instead tell the connected client(s) to check the database for the next job detail.

Might be worth looking into after all. Or, you know, read the RabbitMQ docs properly and fix the problem there. 😉