[ACM] Optimization / caching

[jalvz]

Goal: reduce requests to Kibana as much as possible.
Main risk comes from RUM agent.
This task should include a small benchmark.

Requires #2095

[simitt]

Initial thoughts around cache handling for ACM:

Step 1:

Implement a simple LRU cache with fixed size or a dynamically sized cache with expiring keys, configurable via apm-server.yml.
Cache granularity: service.name + service.env, as those are the currently supported query parameters.

Currently ACM can only be used from authenticated agents (backend agents) when token is configured.
If APM Server is not token secured but publicly reachable this generally leads to the same risks as for backend Intake API (malicious requests, no rate limiting), but adds risk for overloading Kibana.

As long as RUM agents don't support ACM, this seems ok, as we provide our users a way of safely using this endpoint by using a token, and the unsecured setup is not recommended.

Step 2:

Introduce dedicated ACM endpoint for RUM agents or make use of User-Agent information to differentiate RUM and backend agents. This allows to force token usage for backend agents while applying rate limiting for RUM agent requests.

We need to provide a solution where users can enable ACM for RUM without being forced to be open to above mentioned risk by enforcing to disable token. An ACM endpoint used by the RUM agent must be at least as secure as the Intake API endpoint for RUM. The RUM Intake endpoint does not support authentication at the moment, but rate limiting per IP is applied by default.

User-Agent information can easily be faked, but as soon as any non RUM information is provided, the request must provide the token to authenticate (if authentication is enabled).

Applying RUM rate limiting for all requests would be feasible but confusing for two reasons: the config options are nested under rum, and they are not applied to the backend Intake API.

Step 2b:

Implement separated caches for backend and RUM agents, to avoid dropping backend keys from the cache when cache size doesn't fit amount of RUM queries.

This is important, as bad cache config values or malicious RUM requests could overload Kibana. If no cache information for backend agents is stored and Kibana is not reachable, backend agents could not be served. While only static information will be returned with proper caching, in case Kibana is not reachable, this still improves situation for newly started or added backend agents (getting outdated config vs. getting no config from server).

How can we keep config easy for two different caches?
Use a dedicated cache config setting for backend agents. Rename rum.event_rate to rum.cache (in a backwards compatible way) and use the same settings for rate limit and config cache. Maybe provide an option rum.event_rate.enabled to turn off load balancing explicitly.

[simitt]

Update to above proposal (2b): Dedicated cache size setting would make more sense, as the number of unique parallel users will be a different dimension than the number of unique services using the RUM agent.

[graphaelli]

Update: we should not expect to support CM for RUM in this first iteration.