Ability to provide New Relic agent configuration via the User Provided Service payload

[davidehringer]

The current implementation of the New Relic framework, which only allows for license key to be configured, doesn’t provide enough out of the box configurability for many deployments without forking the buildpack.

Scenarios

There are two categories of configurations that we would like to apply:

Application or Organization/Space Specific

There are various parameters we may need to apply for specific app or groups of apps (via org or space boundaries). A common example of this is applying "labels" to categorize applications. (I believe this example is similar to the tier name configuration recently added to the AppDynamics framework).

Site-wide

The standard example of this is the requirement to configure a proxy server. This could be configured in resources/new_relic_agent/newrelic.yml via a buildpack fork. But we are trying to avoid forking the buildpack for very simple configurations like this because maintaining a fork can represent maintenance burden we would like to avoid.

Proposed Implementation

While there are very common examples of configuration parameters, providing a defined set of configurable parameters runs the risk of endlessly adding special support for specific parameters. We should be able to provide a general configuration mechanism that allows for configuring any value in the newrelic.yml. In the Java buildpack, the NewRelicAgent currently leverages the system property override mechanism to configure the New Relic agent.

You can override any setting in the newrelic.yml file by setting a system property. The system property corresponding to a given setting in the config file is setting name prefixed bynewrelic.config. For example, system property for the log_level setting is newrelic.config.log_level.
For settings nested in stanzas, prepend the stanza name to the setting name. For example, the system property for the enabled setting in the transaction_tracer stanza is newrelic.config.transaction_tracer.enabled.

Leveraging this feature and by adding support for additional credentials payload values could provide a generic way to support configurability. The algorithm could be the following. For any value found in the credentials payload, add a system property for that entry. For example, the credentials payload of

"credentials": {
  “license_key": “xxxxx”,
  “proxy_host": “my-proxy.com",
  “proxy_port": “80",
  “labels": “Data Center: Primary, Environment: Performance, Business Unit: My BU"
}

the following system properties would be added:

-Dnewrelic.config.license_key=“xxxxx”
-Dnewrelic.config.proxy_host=“my-proxy.com”
-Dnewrelic.config.proxy_port=“80”
-Dnewrelic.config.labels=“Data Center: Primary, Environment: Performance, Business Unit: My BU”

It appears as though this could be added to the NewRelicAgent.release method in a fairly straightforward manner.

[cgfrost]

Issue created in tracker, thanks for raising. https://www.pivotaltracker.com/story/show/92119348

[imod]

👍 this would be very helpful!

[davidehringer]

Also related to this but potentially items that could be raised as their own issues, there are a few additional considerations. In NewRelicAgent.release it sets the logs dir as well as the app name.
.add_system_property('newrelic.config.app_name', "#{application_name}") .add_system_property('newrelic.config.log_file_path', logs_dir)

The way logs_dir is set, the agent logs basically go nowhere since loggregator doesn't monitor files. This configuration should be replaced with -Dnewrelic.config.log_file_name=STDOUT so that the New Relic agent logs can be processed like all other app logs.

Using #{application_name} to set the New Relic application can result in the agent reporting metrics under the wrong name when cf rename is used. Because this is the common pattern for doing blue/green deployments in CF, it can be an issue.

I think the best solution for these default values is to allow them to be overridden. For example, if the user provides "-Dnewrelic.config.app_name=xxx" in JAVA_OPTS, then that value should be used instead of #{application_name}.

[tedpennings]

👍

One suggestion on the log topic: it would be really useful to be able to set the log level. Troubleshooting issues often requires debug-level agent logs, which is an extra configuration to set.

[cgfrost]

I'm working on the original request at the moment, I expect something will be in master next week. As for the logging changes. I have raised a different tracker ticket to look in to it.
https://www.pivotaltracker.com/story/show/98297746

[sks]

Thanks @cgfrost for this enhancement, really helpful,

Can we at the same time have a flexible way around setting newrelic.config.app_name ?
configuration['app_name'] = @application.details['application_name']

In case the users have a blue green deployment, the application_name always changes and it screws up the new relic dashboard .

I tried adding JAVA_OPTS: -Dnewrelic.config.app_name=my-appname to the manifest.yml, but it gets overridden 😿

What is the preferred way to do this ?

[cgfrost]

Normally that would be fine but the New Relic support is manually setting the app_name. The only place it can be provided is through the credentials of the New Relic service. If this isn't sufficient to support your deployment process then please can you raise another issue on us. I suspect we will have to find a third way for the application name to be provided that over rides the existing mechanism.