Broken queue causes various problems, need to report in self-check

[joshrnoll]

Description

Sometimes, the pull/push mirrors stopped synchronizing.

Gitea Version

All

---

(Edited by wxiaoguang: only keep the problem description, the workaround is in the following comment)

[wxiaoguang]

Maybe this could help your case: #33862 (comment)

Does the queue system work? Check it from admin panel -> monitor -> queue /-/admin/monitor/queue
If the queue system gets stuck, you need to click "Remove All" to clean the corrupted data

[joshrnoll]

This fixed it!

For those who stumble on this in the future:

Go to Site Administration --> Monitoring --> Queues

Click on Review / Add Workers next to the mirror queue, then click Remove all (I had 9 in the queue before clearing)

After doing this I initially received a permission denied error, but this was because I had deleted and recreated my remote repo without recreating the PAT:

Once I recreated my PAT, I got a successful sync again.

@wxiaoguang thanks for the help! Is this a bug? Should I leave this issue open or go ahead and close?

[wxiaoguang]

It is a bug (explained here #33862 (comment))

But it's difficult to fix.

The best approach we could do at the moment is to add the "queue check" to "self check" page to remind users .....

So I think we can leave this open until we have a good "self check" prompt

[wxiaoguang]

By the way, the corruption only occurs in "levelqueue". So you could switch to "redis" queue which won't get stuck 😆

[TheFox0x7]

Would it be possible to replace levelqueue with something that's less prone to failing?

I've been eyeing nats but it's a bit harder to integrate it into queues (with persistence that is) than I originally thought, however it can be embedded into the binary which is why I find worth considering.

[wxiaoguang]

Would it be possible to replace levelqueue with something that's less prone to failing?

No idea at the moment. Gitea's queue has some special behaviors like "unique queue". IMO making levelqueue could self-heal is a quick approach.