Disabling monitoring of system actors has no effect on Live Actors and Actor Start/Stop panels

[object]

Finally we can see our customized F# actor type names (thanks for fixing it so quickly), so we can see our actor types in "Live Actors by Type" and "Actor Starts and Stops" panels. But the panels show mostly system actors, even though monitoring of them is disabled with the following code:

        builder
            .SetMonitorUserActors(true)
            .SetMonitorSystemActors(false)
            .SetMonitorEventStream(true)
            .SetMonitorMailboxDepth(false)

For example, "Live Actors by Type" dashboard looks like this:

Akka.Remote.Transport.ProtocolStateActor 112
Akka.Remote.EndpointReader 112
Akka.Remote.EndpointWriter 112
Akka.Remote.ReliableDeliverySupervisor 110
upload-globalconnect-file 71
upload-globalconnect-file-eventpublisher 71
upload-mediaset-controller 52
Akka.Cluster.Sharding.Shard 30

Only 3 of 8 actor types represent our own actors, the rest are system actors which are of no interest for us to collect metrics for.
Shouldn't SetMonitorSystemActors(false) disable their monitoring so they don't appear in dashboards? If not, would it be possible to support monitor filtering, so we can configure what actor types should be excluded?

[Aaronontheweb]

Hi @object - looks like the issue here is the Akka.NET controls how those actorstarted et al metrics are emitted. The MonitorSystemActors stuff has historically been used just to control Phobos emissions. We can just filter that on the Phobos side so you don't see any of them appear in your charts either. I'll get on that today.

[Aaronontheweb]

This is fixed in our latest dev build but we're going to test it in our lab first

[Aaronontheweb]

Did some testing of this and it still needs more time to cook - we're going to have to make a patch to Akka.NET in order to have this work consistently. In this particular instance, shutting off all /system actor metrics also big-footed /system actors that we want metrics for, such as Akka.Cluster.Sharding and its entities.

[Aaronontheweb]

We basically need to push the decision about which metrics get recorded to the edges (the actors themselves), not the accumulator we use to produce the metric.

[Aaronontheweb]

This issue has been resolved, properly in Phobos 2.8.1 https://phobos.petabridge.com/articles/releases/RELEASE_NOTES.html

[object]

@Aaronontheweb Thanks for the quick update. I have rebuilt our services with Akka 1.5.36 and Phobos 2.8.1. I no longer see in the dashboard some of the system actors that were listed earlier, but other actors from Akka namespace are there. Here's what I see now:

Akka.Persistence.Journal.AsyncWriteJournal+Resequencer 1127618
Akka.Persistence.Extras.PersistenceSupervisor 113 (this one is a part of Akka Extras packages)
Akka.Cluster.Sharding.ShardRegion 28
Akka.Cluster.Sharding.Shard 21
Akka.Cluster.Tools.PublishSubscribe.Internal.Topic 15

How is actor classified as a system actor? By namespace or from a pre-defined list? Can the decision be made based on a namespace?

[Aaronontheweb]

It's classified by where the actor is on the hierarchy - actors that fall under the /system hierarchy have both metrics and tracing disabled by default. The exceptions to that are some of the Akka.Cluster.Sharding actors and the DistributedPubSub actors.

We can probably disable metrics on those by default too - it's just that normally we treat the sharding / distributed pub sub actors like they are /user actors because users have requested traceability on there. But, we do have the ability to tell them "trace like /user actors but do metrics like /system actors"

[Aaronontheweb]

The one of your list that makes me the most interested are the Akka.Persistence.Journal.AsyncWriteJournal+Resequencer - that makes me wonder if that's actually an Akka.NET bug.

[Aaronontheweb]

Well I'll be damned, that is an Akka.NET bug for certain: https://github.com/akkadotnet/akka.net/blob/759e93f55f66e79d8356f6ad1526144126d0b479/src/core/Akka.Persistence/Journal/AsyncWriteJournal.cs#L84

How on earth did you end up with 100k of them though?

[object]

It's a first time I see Akka.Persistence.Journal.AsyncWriteJournal+Resequencer in that list, and I should have seen it earlier because of its huge number of instances. That must certainly be a bug. It can't be that high. We don't have so many live actors, and only a fraction of them are persistent.