Agent remote configuration

[jalvz]

Overview

Following #4 , agents need to be able to poll apm-server for configuration changes received upstream from Kibana, apply them, and log the result with status (success | failure), failure cause (if any), timestamp, setting name and value.

We will start providing support for TRANSACTION_SAMPLE_RATE.

Requirements

At minimum, agents need to agree on:

• name and default value for config polling interval setting (RUM can have a different default).
• exact message pattern and log level.
• top 2-5 settings that should follow sampling rate, in order of priority.

APM Server API

• Server will expose a /config/v1/agents endpoint, for agents to GET with a service.name URL query parameter (required), and service.environment (optional)

• Agents might send a request with a If-None-Match header, to which Server will respond with a 304 - not modified response; or with 200, a response body with the configuration, and an Etag header.

• Example
  curl -v -H "If-None-Match:1" "http://localhost:8200/config/v1/agents?service.environment=prod&service.name=opbeans"

*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8200 (#0)
> GET /config?service.environment=prod&service.name=opbeans HTTP/1.1
> Host: localhost:8200
> User-Agent: curl/7.61.0
> Accept: */*
> If-None-Match:1
> 
< HTTP/1.1 200 OK
< Cache-Control: max-age=0
< Content-Type: application/json
< Etag: 2
< Date: Thu, 11 Apr 2019 14:08:32 GMT
< Content-Length: 27
< 
{
  "transaction_sample_rate": 0.7
}

Update 23/04

Configuration settings are taken from the environment variable names, without the ELASTIC_APM_ prefix, and lower case.

Update 06/05

As per comment #76 (comment), the Server will accept query parameters both in the URL and in the body of a POST request.
If different values for the same attribute are provided as POST and GET, request will be 400-rejected; different attributes will be joint.

Other notes that slipped in the initial description:

• As pointed out in Agent remote configuration #76 (comment), agents should also align the error handling behaviour. For instance, if a config update can't be applied, should fallback to the last good value, to the agents default value, or to the value that the process started with?

• Regarding service.environment, if none is passed in the query, only config updates without service environment will match. Likewise, if one is passed, only config updates with that value will match (and not config updates without value). In other words, a missing service environment is treated like any other (with a value of "" if you want to see it that way).

Update 29/05

• The Server will send all attributes as strings.

Update 01/07

• The Server caches the agent configuration for 30s by default (changeable via apm-server config) and sets the expiration time via Cache-Control: max-age header in every successful response. For failing agent requests the header will be set with a max-age: 300 (5 mins) since querying again after 30s doesn't make sense. Decision was made to set to 5 mins instead of not setting or setting to 0 so agents don't need to put their own logic and can differentiate between server not supporting remote config and failures. More details on this in [ACM] Optimization / caching apm-server#2220.

Updates for 7.3

• Return 403 if ACM not enabled in APM Server (apm-server.kibana.enabled: false) (Fix config handler apm-server#2386)
• Enclose Etag header in double quotes and expect the same for the If-None-Match header (Enclose etag and IfNoneMatch header in double quotes. apm-server#2407)
• Support reconnecting to Kibana client if not available on startup (Ensure Kibana client reconnects. apm-server#2421)
• Return error response bodies in json if supported and return minimal error response body if no security token is used for communicating with the APM Server (Ensure Kibana client reconnects. apm-server#2421)
• Return 200 with empty body instead of 404 if configuration has not been found ([ACM] Return 200 instead of 404 for missing config. apm-server#2458)
• Create Etag from _source.settings and return flattened key:value pairs in case nested config options are stored ([ACM] Create Etag from settings struct. apm-server#2441).

Status

@elastic/apm-agent-devs please link your implementation issues

• .NET - Implement remote (AKA central) configuration apm-agent-dotnet#352
• Go - Remote config apm-agent-go#512
• Java - Remote config apm-agent-java#640
• Node.js - Add ability to change agent sampleRate from within Kibana apm-agent-nodejs#1125
• Python - Implement remote configuration  apm-agent-python#498
• Ruby - Support remote configuration apm-agent-ruby#446
• RUM - Support fetching agent configuration from the APM Server (Central config management) apm-agent-rum-js#253

Let me know if you have any questions.

[felixbarny]

Are there plans to create configurations for a more fine-grained subset of agents, based on more than service_name and service_environment possible even via global labels? If so, it might make sense to send the metadata along with the request.

[SergeyKleyman]

A minor comment: the name for sampling rate option is "TRANSACTION_SAMPLE_RATE" (https://docs.google.com/spreadsheets/d/1JJjZotapacA3FkHc2sv_0wiChILi3uKnkwLTjtBmxwU/) so I would suggest to use "transaction_sample_rate" in response JSON for consistency.

[SergeyKleyman]

Should agents be aligned on other aspects of error handling and not just error reporting? For example, if some of the options fail to pass validation should the agent reject the whole configuration or should the agent use the valid options and find some replacements for the invalid ones (use default values, etc.)

[jalvz]

name for sampling rate option is "TRANSACTION_SAMPLE_RATE"

thanks @SergeyKleyman , updated with a note.

Should agents be aligned on other aspects of error handling

Yes, good point. I did highlight error reporting because it might have an impact in Kibana (eg. someone could create dashboards based on it), so I think it is important for the feature as a whole.
You should discuss error handling and align to the the extent of what is reasonable, but it is essentially up to you.

Are there plans to create configurations for a more fine-grained subset of agents

It was discussed briefly. We can ado that, yes, but it doesn't have the highest priority atm. Any ideas here ofc are welcome.

[felixbarny]

It was discussed briefly. We can ado that, yes, but it doesn't have the highest priority atm. Any ideas here ofc are welcome.

Sure, we don't have to start with that right away. My point is that even if we want to add support for it later, we should already start sending the metadata instead of query parameters now.

[axw]

Sending metadata instead of query parameters does seem like it would make things more extensible. A newer server could start taking advantage of metadata agents already send, without any upgrade of agents. Is this difficult for anyone to implement?

[hmdhk]

At least for the RUM agent being able to get the configuration without sending a body is preferable. Furthermore I think the extensibility provided by sending metadata is not very useful when the available configuration options is limited to a small number as is the case for the RUM agent.

[jalvz]

We can accept both, if that makes happy everyone.

[axw]

@jalvz that sounds fine to me, if it's not too onerous on the server.

Could we just say that you can specify any metadata as a query parameter, using the same JSON path? e.g. where a backend agent might use

GET /config
{
  "service": {
    "name": "foo",
    "environment": "bar"
  }
}

the RUM agent would use

GET /config?service.name=foo&service.environment=bar

i.e. with dots instead of _, so we can directly map the query params to a nested object in the metadata. Then, for completeness, we would state that query parameters would be overlaid on the request body (if any).

[jalvz]

@elastic/apm-agent-devs can I get you to comment on:

top 2-5 settings that should follow sampling rate

Some interesting ones are LOG_LEVEL, METRICS_INTERVAL, SPAN_FRAMES_MIN_DURATION, TRANSACTION_MAX_SPANS, SERVER_TIMEOUT, INSTRUMENT... what would you like to see next?

[felixbarny]

I'd like to see that users can enter any configuration options via text input.