[receiver/mongodbatlas] Add Events API Support

[schmikei]

Component(s)

receiver/mongodbatlas

Is your feature request related to a problem? Please describe.

Looking to add to the receiver to start collecting events from this API. This endpoint contains a multitude of types of information about Host Restarts, Replica set elected a new primary, Alert configuration disabled , Network: Bytes In has gone above 5000 bytes (avg/sec), User joined the project, etc.

I feel like this type of data would be useful for end users to use this data to track rapid restarts and potentially create alerts.

Describe the solution you'd like

I think a good solution for this is to add another section of configuration along with alerts, logs, and metrics which could be called events.

I also think it nice to include a project filtering mechanism in the config along with an event type filtering on the type of event, by default all should be collected.

Example Configuration:

receivers:
  mongodbatlas:
    public_key: <redacted>
    private_key: <redacted>
    events:
      enabled: true
      # only collect AUTH_LOGIN_FAIL and HOST_RESTARTED events
      types: [AUTH_LOGIN_FAIL, HOST_RESTARTED]
      projects:
      - name: Project 0
        include_clusters: [Cluster0]
      poll_interval: 1m
    # use of a storage extension is recommended to reduce chance of duplicated events
    storage: file_storage

Describe alternatives you've considered

I thought about including this in the alerts portion but ultimately decided against it because not all these events are "alerts" but more info level information such as user additions, host restarts, etc.

Additional context

I'm happy to start working on this and willing to be assigned if the feature is acceptable. Happy to hear further suggestions about configuration/overall usability.

[github-actions]

Pinging code owners:

• receiver/mongodbatlas: @zenmoto

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[schmikei]

@zenmoto @djaglowski do you guys have any thoughts about adding this functionality?

[djaglowski]

Seems reasonable to me.

Would there be any sense to include this in the logs section though? Events are basically logs, so I think it makes some sense from that perspective, but I'm sure it comes down to whether or not the configurations would overlap nicely.

I also have to say, I really dislike the enabled: true pattern we wound up with on this receiver. The presence or absence of the section should be sufficient to define that. It's a bit of an aside, but I think we should consider whether it's reasonable to move away from that before adding another instance of the pattern.

[schmikei]

Would there be any sense to include this in the logs section though? Events are basically logs, so I think it makes some sense from that perspective, but I'm sure it comes down to whether or not the configurations would overlap nicely.

I believe the format of the data is fairly different but in essence I agree these are both logs and could/should be configurable in a logs context.

Comparison

Normal Log

{
  "t": {
    "$date": "2022-09-11T18:53:14.675+00:00"
  },
  "s": "I",
  "c": "NETWORK",
  "id": 22944,
  "ctx": "conn35107",
  "msg": "Connection ended",
  "attr": {
    "remote": "192.168.248.2:52066",
    "uuid": "d3f4641a-14ca-4a24-b5bb-7d7b391a02e7",
    "connectionId": 35107,
    "connectionCount": 33
  }
}

Event JSON representation

{
"created": "2018-06-04T19:19:00Z",
"eventTypeName": "HOST_NOW_PRIMARY",
"groupId": "5b478b3afc4625789ce616a3",
"hostname": "example-shard-0-abc0p.mongodb.net",
"id": "5b478c2562c892f9824cd990",
"links": [
{
"href": "https://cloud.mongodb.com/api/atlas/v1.0/groups/5b478b3afc4625789ce616a3/events/5b478c2562c892f9824cd990",
"rel": "self"
}
],
"port": 27017,
"replicaSetName": "example-shard-0"
}

I think the configuration parameters needed for the events to be functional:

• public_key (from top level config)
• private_key
• Projects List (default to collect all)
• List of Event Types to collect (default to collect all)

So perhaps if we decided to incorporate it into the current logs configuration it could look like this

receivers:
  mongodbatlas:
    public_key: <some-key>
    private_key: <some-key>
    logs:
      enabled: true # can revisit potentially
      projects: 
        - name: "project 1"
          collect_audit_logs: true
          events: 
             types: [HOST_RESTARTED, etc.]

However it should be mentioned that the ProjectConfig struct is also shared among both alerts and logs although the collect_audit_logs is only relevant for logs collection, but I do overall think that this config probably reads better than the initial one where they are separated.

[djaglowski]

Ok, my inclination would be to just use a new config struct then. Ideally none of these distinct mechanisms for collection should share configs unless one is a clean superset of the other.

[zenmoto]

I agree with keeping it in an events block. The log SIG has been talking about adding support for events- it's not ready yet, but these seem like they fit the criteria (named type, structured payload), so keeping them a little separate seems better to me. I also agree with @djaglowski about the enabled: true for these events- it seems like defaulting to an empty list of events is clearer and does the same thing.