[APM] Support for deriving service version annotations per service node

[tobiasstadler]

Describe the feature:
It would be nice, if the APM app could optionally derive the service version annotations for each service node separately (see https://github.com/elastic/kibana/blob/master/x-pack/plugins/apm/server/lib/services/annotations/get_derived_service_annotations.ts). It should be configurable if the the derivation is per service node or globally.

Describe a specific use case for the feature:
We are using static names for service instances and I would like to see in the service overview at which time a version was deployed on which host.

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[ogupte]

It should be configurable if the the derivation is per service node or globally.

I don't think this needs to be configurable, it seems useful to always show service version annotations per node for the selected service.

[tobiasstadler]

It should be configurable if the the derivation is per service node or globally.

I don't think this needs to be configurable, it seems useful to always show service version annotations per node for the selected service.

I was thinking of containerized environments where the hostname changes with every deployment and every restart of the application. In such an environment it might not make sense to display an annotation for every restart of the application.

[ogupte]

I was thinking of containerized environments where the hostname changes with every deployment and every restart of the application. In such an environment it might not make sense to display an annotation for every restart of the application.

I feel like container restarts are something we'd want to show in the service annotations anyways. Though we might want to display it differently. like if there were no events with the same service.node.name within the selected time range, we'd show a "service start" annotation which includes the version and node name. @formgeist, what do you think about something like this?

[tobiasstadler]

I am fine either way.