Quoting @richard-ramos Discord conversation:

• There seems to be an issue with filter in which messages are pushed to the clients with a noticeable delay.
• Up until yesterday, there was a job that automagically deployed changes in nwaku master to shards.fleet. DB got broken because of that
• QA mentioned that communities are not loading the message history (currently debugging this)

Quoting @chaitanyaprem from Discord conversation:

As of now, i see a lot of subscribe requests when i start my desktop app (but that could just be to different contentTopics) and unsubscribe only per pubsubTopic (which could just include all contentTopics which were subscribed).

Unable to debug further because of not logging contentTopics and not having an API to confirm what FilterSubscriptions are active as of now.

Maybe it would be good to have some Admin API's to query list of Filter Subscriptions and similar useful API's for monitoring other protocols as well.This may help node operators and also in turn Waku team to debug issues.

For now, I will try to figure out another way what could be causing this issue. And at the same time recommend to use unsubscribeAll while shutting down or existing status-go.

Couple of more observation we had noticed during this session:
In case peers are subscribed and are not available/alive , for each filterPush we try to dial the peers which seems like a bad strategy. If peers are down for few mins, then this leads to a lot of dials from the filterFullNode(which doesn't look optimal).

Secondly, the idle/inactive period after which a filterFullNode considers a subscription to be invalid may need to be reduced to maybe 30 mins or so(not sure what is the optimal value here at this point). But, having such idle subscriptions hang on a longer time could just be a minor attack vector on a service-node. An attacker could just spawn peers randomly and keep hitting with subscriptions and not unsubscribe leaving filterFullNode wasting resources just dialing peers for each message. Not sure if this is a big issue, but something to think of.

QA Test Scenario Description

Scenario 1
1.
2.
Scenario 2
1.
2.