Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add add_observer_metadata processor #11394

Merged
merged 19 commits into from
Apr 24, 2019
Prev Previous commit
Next Next commit
Further
  • Loading branch information
andrewvc committed Apr 23, 2019
commit c659cc6a4240d11a5da588a84262ff3b44f025b6
3 changes: 2 additions & 1 deletion heartbeat/heartbeat.yml
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,8 @@ output.elasticsearch:

#================================ Processors =====================================
processors:
- add_observer_metadata: ~
- add_observer_metadata:
netinfo.enabled: true


#================================ Logging =====================================
Expand Down
1 change: 1 addition & 0 deletions libbeat/cmd/instance/imports.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ import (
_ "github.com/elastic/beats/libbeat/processors/add_host_metadata"
_ "github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata"
_ "github.com/elastic/beats/libbeat/processors/add_locale"
_ "github.com/elastic/beats/libbeat/processors/add_observer_metadata"
_ "github.com/elastic/beats/libbeat/processors/add_process_metadata"
_ "github.com/elastic/beats/libbeat/processors/communityid"
_ "github.com/elastic/beats/libbeat/processors/dissect"
Expand Down
70 changes: 69 additions & 1 deletion libbeat/docs/processors-using.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,7 @@ The supported processors are:
* <<add-docker-metadata,`add_docker_metadata`>>
* <<add-fields, `add_fields`>>
* <<add-host-metadata,`add_host_metadata`>>
* <<add-observer-metadata,`add_observer_metadata`>>
* <<add-kubernetes-metadata,`add_kubernetes_metadata`>>
* <<add-labels, `add_labels`>>
* <<add-locale,`add_locale`>>
Expand Down Expand Up @@ -1173,7 +1174,7 @@ It has the following settings:


The `add_host_metadata` processor annotates each event with relevant metadata from the host machine.
The fields added to the event are looking as following:
The fields added to the event are look like following:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The fields added to the event are look like following:
The fields added to the event look like following:


[source,json]
-------------------------------------------------------------------------------
Expand Down Expand Up @@ -1205,6 +1206,73 @@ The fields added to the event are looking as following:
}
-------------------------------------------------------------------------------

[[add-observer-metadata]]
=== Add Observer metadata

beta[]

[source,yaml]
-------------------------------------------------------------------------------
processors:
- add_observer_metadata:
netinfo.enabled: false
cache.ttl: 5m
geo:
name: nyc-dc1-rack1
location: 40.7128, -74.0060
continent_name: North America
country_iso_code: US
region_name: New York
region_iso_code: NY
city_name: New York
-------------------------------------------------------------------------------

It has the following settings:

`netinfo.enabled`:: (Optional) Default false. Include IP addresses and MAC addresses as fields observer.ip and observer.mac

`cache.ttl`:: (Optional) The processor uses an internal cache for the observer metadata. This sets the cache expiration time. The default is 5m, negative values disable caching altogether.

`geo.name`:: User definable token to be used for identifying a discrete location. Frequently a datacenter, rack, or similar.

`geo.location`:: Longitude and latitude in comma separated format.

`geo.continent_name`:: Name of the continent.

`geo.country_name`:: Name of the country.

`geo.region_name`:: Name of the region.

`geo.city_name`:: Name of the city.

`geo.country_iso_code`:: ISO country code.

`geo.region_iso_code`:: ISO region code.


The `add_geo_metadata` processor annotates each event with relevant metadata from the observer machine.
The fields added to the event look like the following:

[source,json]
-------------------------------------------------------------------------------
{
"host":{

"ip": ["192.168.0.1", "10.0.0.1"],
"mac": ["00:25:96:12:34:56", "72:00:06:ff:79:f1"],
"geo": {
"continent_name": "North America",
"country_iso_code": "US",
"region_name": "New York",
"region_iso_code": "NY",
"city_name": "New York",
"name": "nyc-dc1-rack1",
"location": "40.7128, -74.0060"
}
}
}
-------------------------------------------------------------------------------

[[dissect]]
=== Dissect strings

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ import (
)

func init() {
processors.RegisterPlugin("add_observer_metadata", newObserverMetadataProcessor)
processors.RegisterPlugin("add_observer_metadata", New)
}

type observerMetadata struct {
Expand All @@ -52,7 +52,7 @@ const (
processorName = "add_observer_metadata"
)

func newObserverMetadataProcessor(cfg *common.Config) (processors.Processor, error) {
func New(cfg *common.Config) (processors.Processor, error) {
andrewvc marked this conversation as resolved.
Show resolved Hide resolved
config := defaultConfig()
if err := cfg.Unpack(&config); err != nil {
return nil, errors.Wrapf(err, "fail to unpack the %v configuration", processorName)
Expand Down
7 changes: 6 additions & 1 deletion libbeat/processors/processor.go
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,12 @@ func New(config PluginConfig) (*Processors, error) {

gen, exists := registry.reg[actionName]
if !exists {
return nil, errors.Errorf("the processor action %s does not exist", actionName)
var validActions []string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is this change related?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made a mistake in earlier versions of this and found this debugging info useful. Glad to move it to a new PR if you feel it's worthwhile.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

for k := range registry.reg {
validActions = append(validActions, k)

}
return nil, errors.Errorf("the processor action %s does not exist. Valid actions: %v", actionName, strings.Join(validActions, ", "))
}

actionCfg.PrintDebugf("Configure processor action '%v' with:", actionName)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ import (
"github.com/elastic/beats/libbeat/processors/add_host_metadata"
"github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata"
"github.com/elastic/beats/libbeat/processors/add_locale"
"github.com/elastic/beats/libbeat/processors/add_observer_metadata"
"github.com/elastic/beats/libbeat/processors/add_process_metadata"
"github.com/elastic/beats/libbeat/processors/communityid"
"github.com/elastic/beats/libbeat/processors/dissect"
Expand All @@ -44,6 +45,7 @@ var constructors = map[string]processors.Constructor{
"AddFields": actions.CreateAddFields,
"AddHostMetadata": add_host_metadata.New,
"AddKubernetesMetadata": add_kubernetes_metadata.New,
"AddObserverMetadata": add_observer_metadata.New,
"AddLocale": add_locale.New,
"AddProcessMetadata": add_process_metadata.New,
"CommunityID": communityid.New,
Expand Down