Skip to content

Commit

Permalink
[receiver/googlecloudmonitoringreceiver] Adding Google Cloud monitori…
Browse files Browse the repository at this point in the history
…ng receiver (#34015)

**Description:** Adding Google Cloud Monitoring receiver

Closes #33762

**Testing:** Test cases for configuration added

**Documentation:** README file added for describing the component
  • Loading branch information
abhishek-at-cloudwerx authored Aug 12, 2024
1 parent 00e1b30 commit 2515c52
Show file tree
Hide file tree
Showing 22 changed files with 712 additions and 0 deletions.
27 changes: 27 additions & 0 deletions .chloggen/googlecloudmonitoringreceiver-phase1.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Use this changelog template to create an entry for release notes.

# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
change_type: new_component

# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
component: googlecloudmonitoringreceiver

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: Adding new component - [Google Cloud monitoring](https://cloud.google.com/monitoring/api/metrics_gcp) receiver to fetch GCP Cloud Metrics and transform to OpenTelemetry compatible format.

# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
issues: [33762]

# (Optional) One or more lines of additional information to render under the primary note.
# These lines will be padded with 2 spaces and then inserted directly into the document.
# Use pipe (|) for multiline entries.
subtext:

# If your change doesn't affect end users or the exported elements of any package,
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
# Optional: The change log or logs in which this entry should be included.
# e.g. '[user]' or '[user, api]'
# Include 'user' if the change is relevant to end users.
# Include 'api' if there is a change to a library API.
# Default: '[user]'
change_logs: [user, api]
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -297,6 +297,7 @@ receiver/windowseventlogreceiver/ @open-teleme
receiver/windowsperfcountersreceiver/ @open-telemetry/collector-contrib-approvers @dashpole @alxbl @pjanotti
receiver/zipkinreceiver/ @open-telemetry/collector-contrib-approvers @MovieStoreGuy @andrzej-stencel @crobert-1
receiver/zookeeperreceiver/ @open-telemetry/collector-contrib-approvers @djaglowski
receiver/googlecloudmonitoringreceiver/ @open-telemetry/collector-contrib-approvers @dashpole @TylerHelmuth @abhishek-at-cloudwerx

testbed/ @open-telemetry/collector-contrib-approvers @open-telemetry/collector-approvers
testbed/mockdatasenders/mockdatadogagentexporter/ @open-telemetry/collector-contrib-approvers @boostchicken
Expand Down
1 change: 1 addition & 0 deletions .github/ISSUE_TEMPLATE/bug_report.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -229,6 +229,7 @@ body:
- receiver/flinkmetrics
- receiver/fluentforward
- receiver/gitprovider
- receiver/googlecloudmonitoring
- receiver/googlecloudpubsub
- receiver/googlecloudspanner
- receiver/haproxy
Expand Down
1 change: 1 addition & 0 deletions .github/ISSUE_TEMPLATE/feature_request.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,7 @@ body:
- receiver/flinkmetrics
- receiver/fluentforward
- receiver/gitprovider
- receiver/googlecloudmonitoring
- receiver/googlecloudpubsub
- receiver/googlecloudspanner
- receiver/haproxy
Expand Down
1 change: 1 addition & 0 deletions .github/ISSUE_TEMPLATE/other.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,7 @@ body:
- receiver/flinkmetrics
- receiver/fluentforward
- receiver/gitprovider
- receiver/googlecloudmonitoring
- receiver/googlecloudpubsub
- receiver/googlecloudspanner
- receiver/haproxy
Expand Down
1 change: 1 addition & 0 deletions .github/ISSUE_TEMPLATE/unmaintained.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,7 @@ body:
- receiver/flinkmetrics
- receiver/fluentforward
- receiver/gitprovider
- receiver/googlecloudmonitoring
- receiver/googlecloudpubsub
- receiver/googlecloudspanner
- receiver/haproxy
Expand Down
1 change: 1 addition & 0 deletions receiver/googlecloudmonitoringreceiver/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
include ../../Makefile.Common
58 changes: 58 additions & 0 deletions receiver/googlecloudmonitoringreceiver/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# Google Cloud Monitoring Receiver

<!-- status autogenerated section -->
| Status | |
| ------------- |-----------|
| Stability | [development]: metrics |
| Distributions | [contrib] |
| Issues | [![Open issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector-contrib?query=is%3Aissue%20is%3Aopen%20label%3Areceiver%2Fgooglecloudmonitoring%20&label=open&color=orange&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Areceiver%2Fgooglecloudmonitoring) [![Closed issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector-contrib?query=is%3Aissue%20is%3Aclosed%20label%3Areceiver%2Fgooglecloudmonitoring%20&label=closed&color=blue&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Areceiver%2Fgooglecloudmonitoring) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@dashpole](https://www.github.com/dashpole), [@TylerHelmuth](https://www.github.com/TylerHelmuth), [@abhishek-at-cloudwerx](https://www.github.com/abhishek-at-cloudwerx) |

[development]: https://github.com/open-telemetry/opentelemetry-collector#development
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
<!-- end autogenerated section -->

The primary objective of the Google Cloud Monitoring Receiver is to gather time series metrics data from all Google services and convert this data into a pipeline format that facilitates further use.

This receiver gets GCP (Google Cloud Platform) metrics from [GCP Monitoring REST API] via the [Google SDK for GCP Metrics] and then convert those timeseries data into OTel Format [Pipeline Data].

[GCP Monitoring REST API]: https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.timeSeries/list
[Google SDK for GCP Metrics]: https://pkg.go.dev/cloud.google.com/go/monitoring/apiv3
[Pipeline Data]: https://pkg.go.dev/go.opentelemetry.io/collector/pdata

## Configuration
The following configuration options are supported:

```yaml
receivers:
googlecloudmonitoring:
collection_interval: 120s
project_id: my-project-id
metrics_list:
- metric_name: "compute.googleapis.com/instance/cpu/usage_time"
delay: 60s
- metric_name: "connectors.googleapis.com/flex/instance/cpu/usage_time"
delay: 60s
```
- `collection_interval` (Optional): The interval at which metrics are collected. Default is 60s.
- `initial_delay` (default = `1s`): defines how long this receiver waits before starting.
- `timeout`: (default = `1m`) The timeout of running commands against the GCP Monitoring REST API.
- `project_id` (Required): The GCP project ID.
- `metrics_list` (Required): A list of services metrics to monitor.

Each single metric can have the following configuration:

- `metric_name` (Required): The specific metric name to collect.
- `delay` (Optional): The delay before starting the collection of metrics for this service. Default is 0s.


## Authentication with Google Cloud

For more details on authentication, refer to the Google Cloud [Application Default Credentials documentation](https://cloud.google.com/docs/authentication/application-default-credentials). Note that if your workload is running on Google Cloud Platform (GCP), the service account credentials will be used automatically without needing to set the environment variable manually.

### Filtering

**Metrics Parameters** :

- A monitoring filter that specifies which time series should be returned. The filter must specify a single metric type. For example: `metric_name: "compute.googleapis.com/instance/cpu/usage_time"`
56 changes: 56 additions & 0 deletions receiver/googlecloudmonitoringreceiver/config.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
// Copyright The OpenTelemetry Authors
// SPDX-License-Identifier: Apache-2.0

package googlecloudmonitoringreceiver // import "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/googlecloudmonitoringreceiver"

import (
"errors"
"fmt"
"time"

"go.opentelemetry.io/collector/receiver/scraperhelper"
)

const minCollectionIntervalSeconds = 60

type Config struct {
scraperhelper.ControllerConfig `mapstructure:",squash"`

ProjectID string `mapstructure:"project_id"`
MetricsList []MetricConfig `mapstructure:"metrics_list"`
}

type MetricConfig struct {
MetricName string `mapstructure:"metric_name"`
Delay time.Duration `mapstructure:"delay"`
}

func (config *Config) Validate() error {
if config.CollectionInterval.Seconds() < minCollectionIntervalSeconds {
return fmt.Errorf("\"collection_interval\" must be not lower than %v seconds, current value is %v seconds", minCollectionIntervalSeconds, config.CollectionInterval.Seconds())
}

if len(config.MetricsList) == 0 {
return errors.New("missing required field \"metrics_list\" or its value is empty")
}

for _, metric := range config.MetricsList {
if err := metric.Validate(); err != nil {
return err
}
}

return nil
}

func (metric MetricConfig) Validate() error {
if metric.MetricName == "" {
return errors.New("field \"metric_name\" is required and cannot be empty for metric configuration")
}

if metric.Delay < 0 {
return errors.New("field \"delay\" cannot be negative for metric configuration")
}

return nil
}
120 changes: 120 additions & 0 deletions receiver/googlecloudmonitoringreceiver/config_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
// Copyright The OpenTelemetry Authors
// SPDX-License-Identifier: Apache-2.0

package googlecloudmonitoringreceiver

import (
"path/filepath"
"testing"
"time"

"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.opentelemetry.io/collector/component"
"go.opentelemetry.io/collector/confmap/confmaptest"
"go.opentelemetry.io/collector/receiver/scraperhelper"

"github.com/open-telemetry/opentelemetry-collector-contrib/receiver/googlecloudmonitoringreceiver/internal/metadata"
)

func TestLoadConfig(t *testing.T) {
cm, err := confmaptest.LoadConf(filepath.Join("testdata", "config.yaml"))
require.NoError(t, err)
factory := NewFactory()
cfg := factory.CreateDefaultConfig()

sub, err := cm.Sub(component.NewIDWithName(metadata.Type, "").String())
require.NoError(t, err)
require.NoError(t, sub.Unmarshal(cfg))

assert.Equal(t,
&Config{
ControllerConfig: scraperhelper.ControllerConfig{
CollectionInterval: 120 * time.Second,
InitialDelay: 1 * time.Second,
},
ProjectID: "my-project-id",
MetricsList: []MetricConfig{
{
MetricName: "compute.googleapis.com/instance/cpu/usage_time",
Delay: 60 * time.Second,
},
{
MetricName: "connectors.googleapis.com/flex/instance/cpu/usage_time",
Delay: 60 * time.Second,
},
},
},
cfg,
)
}

func TestValidateService(t *testing.T) {
testCases := map[string]struct {
metric MetricConfig
requireError bool
}{
"Valid Service": {
MetricConfig{
MetricName: "metric_name",
Delay: 0 * time.Second,
}, false},
"Empty MetricName": {
MetricConfig{
MetricName: "",
Delay: 0,
}, true},
"Negative Delay": {
MetricConfig{
MetricName: "metric_name",
Delay: -1 * time.Second,
}, true},
}

for name, testCase := range testCases {
t.Run(name, func(t *testing.T) {
err := testCase.metric.Validate()
if testCase.requireError {
require.Error(t, err)
} else {
require.NoError(t, err)
}
})
}
}

func TestValidateConfig(t *testing.T) {
validMetric := MetricConfig{
MetricName: "metric_name",
Delay: 0 * time.Second,
}

testCases := map[string]struct {
metricsList []MetricConfig
collectionInterval time.Duration
requireError bool
}{
"Valid Config": {[]MetricConfig{validMetric}, 60 * time.Second, false},
"Empty Services": {nil, 60 * time.Second, true},
"Invalid Service in Services": {[]MetricConfig{{}}, 60 * time.Second, true},
"Invalid Collection Interval": {[]MetricConfig{validMetric}, 0 * time.Second, true},
}

for name, testCase := range testCases {
t.Run(name, func(t *testing.T) {
cfg := &Config{
ControllerConfig: scraperhelper.ControllerConfig{
CollectionInterval: testCase.collectionInterval,
},
MetricsList: testCase.metricsList,
}

err := cfg.Validate()
if testCase.requireError {
require.Error(t, err)
} else {
require.NoError(t, err)
}
})
}
}
6 changes: 6 additions & 0 deletions receiver/googlecloudmonitoringreceiver/doc.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
// Copyright The OpenTelemetry Authors
// SPDX-License-Identifier: Apache-2.0

//go:generate mdatagen metadata.yaml

package googlecloudmonitoringreceiver // import "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/googlecloudmonitoringreceiver"
49 changes: 49 additions & 0 deletions receiver/googlecloudmonitoringreceiver/factory.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
// Copyright The OpenTelemetry Authors
// SPDX-License-Identifier: Apache-2.0

package googlecloudmonitoringreceiver // import "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/googlecloudmonitoringreceiver"

import (
"context"

"go.opentelemetry.io/collector/component"
"go.opentelemetry.io/collector/consumer"
"go.opentelemetry.io/collector/receiver"
"go.opentelemetry.io/collector/receiver/scraperhelper"

"github.com/open-telemetry/opentelemetry-collector-contrib/receiver/googlecloudmonitoringreceiver/internal/metadata"
)

func NewFactory() receiver.Factory {
return receiver.NewFactory(
metadata.Type,
createDefaultConfig,
receiver.WithMetrics(createMetricsReceiver, metadata.MetricsStability))
}

// createDefaultConfig creates the default exporter configuration
func createDefaultConfig() component.Config {
return &Config{
ControllerConfig: scraperhelper.NewDefaultControllerConfig(),
}
}

func createMetricsReceiver(
_ context.Context,
settings receiver.Settings,
baseCfg component.Config,
consumer consumer.Metrics,
) (receiver.Metrics, error) {

rCfg := baseCfg.(*Config)
r := newGoogleCloudMonitoringReceiver(rCfg, settings.Logger)

scraper, err := scraperhelper.NewScraper(metadata.Type.String(), r.Scrape, scraperhelper.WithStart(r.Start),
scraperhelper.WithShutdown(r.Shutdown))
if err != nil {
return nil, err
}

return scraperhelper.NewScraperControllerReceiver(&rCfg.ControllerConfig, settings, consumer,
scraperhelper.AddScraper(scraper))
}
Loading

0 comments on commit 2515c52

Please sign in to comment.