[receiver/cloudflarereceiver] New Alpha Logs Component cloudflarereceiver #17993

JonathanWamsley · 2023-01-24T16:39:36Z

Description:

Adds a new cloudflare receiver that polls Cloudflare's LogPull Enterprise API.

It uses Cloudflare Authentication headers in the form of a X-Auth-Email and X-Auth-Key combination or an API Token.

Link to tracking Issue: Resolves #17376

Testing:
Unit testing and mocked integration testing were performed.

Documentation: New README.md

Note: Waiting on a sandbox cloudflare env for final testing before exiting from draft pr.

runforesight · 2023-01-24T16:40:51Z

Foresight Summary

Major Impacts

build-and-test-windows duration(4 seconds) has decreased 40 minutes 46 seconds compared to main branch avg(40 minutes 50 seconds).

View More Details

✅ tracegen workflow has finished in 5 minutes 19 seconds (⚠️ 2 minutes 52 seconds more than `main` branch avg.) and finished at 24th Jan, 2023.

Job	Failed Steps	Tests
build-dev	- 🔗	N/A	See Details
publish-latest	- 🔗	N/A	See Details
publish-stable	- 🔗	N/A	See Details

⭕ build-and-test-windows workflow has finished in 4 seconds (40 minutes 46 seconds less than `main` branch avg.) and finished at 14th Feb, 2023.

Job	Failed Steps	Tests
windows-unittest-matrix	- 🔗	N/A	See Details
windows-unittest	- 🔗	N/A	See Details

✅ telemetrygen workflow has finished in 54 seconds (2 minutes 12 seconds less than `main` branch avg.) and finished at 14th Feb, 2023.

Job	Failed Steps	Tests
build-dev	- 🔗	N/A	See Details
publish-latest	- 🔗	N/A	See Details
publish-stable	- 🔗	N/A	See Details

✅ check-links workflow has finished in 1 minute 28 seconds (1 minute 6 seconds less than `main` branch avg.) and finished at 14th Feb, 2023.

Job	Failed Steps	Tests
changed files	- 🔗	N/A	See Details
check-links	- 🔗	N/A	See Details

✅ changelog workflow has finished in 2 minutes 19 seconds (1 minute 24 seconds less than `main` branch avg.) and finished at 14th Feb, 2023.

Job	Failed Steps	Tests
changelog	- 🔗	N/A	See Details

✅ prometheus-compliance-tests workflow has finished in 14 minutes 28 seconds (⚠️ 5 minutes 39 seconds more than `main` branch avg.) and finished at 14th Feb, 2023.

Job	Failed Steps	Tests
prometheus-compliance-tests	- 🔗	✅ 21 ❌ 0 ⏭ 0 🔗	See Details

❌ build-and-test workflow has finished in 35 minutes 7 seconds (30 minutes 29 seconds less than `main` branch avg.) and finished at 14th Feb, 2023. 4 jobs failed. There are 2 test failures.

Job	Failed Steps	Tests
correctness-metrics	- 🔗	✅ 2 ❌ 0 ⏭ 0 🔗	See Details
correctness-traces	- 🔗	✅ 17 ❌ 0 ⏭ 0 🔗	See Details
unittest-matrix (1.18, internal)	- 🔗	✅ 561 ❌ 0 ⏭ 0 🔗	See Details
unittest-matrix (1.19, extension)	- 🔗	✅ 537 ❌ 0 ⏭ 0 🔗	See Details
unittest-matrix (1.18, processor)	- 🔗	✅ 1528 ❌ 0 ⏭ 0 🔗	See Details
unittest-matrix (1.19, processor)	- 🔗	✅ 1528 ❌ 0 ⏭ 0 🔗	See Details
unittest-matrix (1.19, internal)	- 🔗	✅ 561 ❌ 0 ⏭ 0 🔗	See Details
unittest-matrix (1.18, receiver-0)	- 🔗	✅ 2613 ❌ 0 ⏭ 0 🔗	See Details
unittest-matrix (1.18, exporter)	- 🔗	✅ 2455 ❌ 0 ⏭ 0 🔗	See Details
unittest-matrix (1.19, receiver-0)	- 🔗	✅ 2613 ❌ 0 ⏭ 0 🔗	See Details
unittest-matrix (1.19, receiver-1)	- 🔗	✅ 1928 ❌ 0 ⏭ 0 🔗	See Details
unittest-matrix (1.18, other)	Run Unit Tests 🔗	✅ 4613 ❌ 2 ⏭ 0 🔗	See Details
unittest-matrix (1.18, receiver-1)	- 🔗	✅ 1810 ❌ 0 ⏭ 0 🔗	See Details
unittest-matrix (1.19, exporter)	- 🔗	✅ 2110 ❌ 0 ⏭ 0 🔗	See Details
unittest-matrix (1.19, other)	- 🔗	✅ 4508 ❌ 0 ⏭ 0 🔗	See Details
integration-tests	- 🔗	✅ 55 ❌ 0 ⏭ 0 🔗	See Details
setup-environment	- 🔗	N/A	See Details
check-codeowners	- 🔗	N/A	See Details
lint-matrix (receiver-0)	- 🔗	N/A	See Details
lint-matrix (receiver-1)	- 🔗	N/A	See Details
lint-matrix (processor)	- 🔗	N/A	See Details
lint-matrix (exporter)	- 🔗	N/A	See Details
lint-matrix (extension)	- 🔗	N/A	See Details
lint-matrix (internal)	- 🔗	N/A	See Details
lint-matrix (other)	Lint 🔗	N/A	See Details
check-collector-module-version	- 🔗	N/A	See Details
build-examples	- 🔗	N/A	See Details
checks	- 🔗	N/A	See Details
unittest-matrix (1.18, extension)	- 🔗	N/A	See Details
unittest (1.19)	- 🔗	N/A	See Details
unittest (1.18)	Interpret result 🔗	N/A	See Details
lint	Interpret result 🔗	N/A	See Details
cross-compile	- 🔗	N/A	See Details
windows-msi	- 🔗	N/A	See Details
build-package	- 🔗	N/A	See Details
publish-check	- 🔗	N/A	See Details
publish-dev	- 🔗	N/A	See Details
publish-stable	- 🔗	N/A	See Details

✅ load-tests workflow has finished in 18 minutes 38 seconds and finished at 14th Feb, 2023.

Job	Failed Steps	Tests
loadtest (TestIdleMode)	- 🔗	✅ 1 ❌ 0 ⏭ 0 🔗	See Details
loadtest (TestMetric10kDPS\|TestMetricsFromFile)	- 🔗	✅ 6 ❌ 0 ⏭ 0 🔗	See Details
loadtest (TestTraceAttributesProcessor)	- 🔗	✅ 3 ❌ 0 ⏭ 0 🔗	See Details
loadtest (TestTraceNoBackend10kSPS\|TestTrace1kSPSWithAttrs)	- 🔗	✅ 8 ❌ 0 ⏭ 0 🔗	See Details
loadtest (TestBallastMemory\|TestLog10kDPS)	- 🔗	✅ 18 ❌ 0 ⏭ 0 🔗	See Details
loadtest (TestMetricResourceProcessor\|TestTrace10kSPS)	- 🔗	✅ 12 ❌ 0 ⏭ 0 🔗	See Details
loadtest (TestTraceBallast1kSPSWithAttrs\|TestTraceBallast1kSPSAddAttrs)	- 🔗	✅ 10 ❌ 0 ⏭ 0 🔗	See Details
setup-environment	- 🔗	N/A	See Details

✅ e2e-tests workflow has finished in 17 minutes 1 second and finished at 14th Feb, 2023.

Job	Failed Steps	Tests
kubernetes-test	- 🔗	N/A	See Details

🔎 See details on Foresight

^{*You can configure Foresight comments in your organization settings page.}

receiver/cloudflarereceiver/logs.go

atoulme · 2023-01-24T16:49:15Z

receiver/cloudflarereceiver/logs.go

+	l.logger.Debug("starting to poll for Cloudflare logs")
+	storageClient, err := adapter.GetStorageClient(ctx, host, l.storageID, l.id)
+	if err != nil {
+		l.logger.Error("failed to set up storage: %w", zap.Error(err))


should this return the err and stop start instead?

getting storage client returns an error now on start

atoulme · 2023-01-24T16:50:23Z

receiver/cloudflarereceiver/logs.go

+}
+
+func (l *logsReceiver) startPolling(ctx context.Context, _ component.Host) {
+	l.logger.Debug("starting cloudflare receiver in retrieval mode")


Suggested change

l.logger.Debug("starting cloudflare receiver in retrieval mode")

l.logger.Debug("starting cloudflare receiver")

github-actions · 2023-02-08T05:17:29Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

atoulme · 2023-02-12T08:49:14Z

internal/components/components.go

@@ -228,6 +229,7 @@ func Components() (otelcol.Factories, error) {
 		bigipreceiver.NewFactory(),
 		carbonreceiver.NewFactory(),
 		chronyreceiver.NewFactory(),
+		cloudflarereceiver.NewFactory(),


you should check out how to add your receiver now to this list, this has changed.

atoulme · 2023-02-12T08:50:10Z

receiver/cloudflarereceiver/client.go

+	case cfg.Auth.APIToken != "":
+		api, err = cloudflare.NewWithAPIToken(cfg.Auth.APIToken)
+	default:
+		return nil, errInvalidAuthenticationConfigured


can this be done in cfg.Validate() ?

actually, it is done in there - so you don't need to check that here anymore. cfg.Validate() is called before the component is created.

Updated, removed default case

atoulme · 2023-02-12T08:53:14Z

receiver/cloudflarereceiver/factory.go

+
+// addRequiredFields adds fields EdgeEndTimestamp and EdgeResponseStatus which
+// are the logs timestamp and severity status value.
+func addRequiredFields(cfg *Config) {


that's a bit odd, and we try not to mutate config if possible. Any chance this can be done downstream in the receiver?

Thank you for the feedback. I'll talk to my peers about it. We also just got the enterprise cloudflare environment up, so
I'll be able to verify a successful run soon.

atoulme · 2023-02-12T08:58:44Z

receiver/cloudflarereceiver/logs.go

+}
+
+func putStringToMapNotNil(m pcommon.Map, k string, v *string) {
+	if v != nil {


Do you have a test case testing when v is nil?

added TestAddNilValuesToMap.

atoulme

Looking good - some nits but overall ready to accept. Please rebase too.

…d fields when custom fields are supplied

github-actions · 2023-03-01T05:17:48Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

JonathanWamsley · 2023-03-03T21:00:09Z

Closing this pr because LogPush described in this issue seems to be the best way moving forward.

github-actions bot added the cmd/configschema configschema command label Jan 24, 2023

github-actions bot requested review from dmitryax, mx-psi and pmcollins January 24, 2023 16:40

atoulme reviewed Jan 24, 2023

View reviewed changes

receiver/cloudflarereceiver/logs.go Show resolved Hide resolved

atoulme reviewed Jan 24, 2023

View reviewed changes

JonathanWamsley force-pushed the add-cloudflarereceiver branch from 74a193d to f14a393 Compare January 24, 2023 18:12

github-actions bot added cmd/otelcontribcol otelcontribcol command cmd/oteltestbedcol labels Jan 24, 2023

JonathanWamsley force-pushed the add-cloudflarereceiver branch 2 times, most recently from 3659753 to 3dcd8df Compare January 24, 2023 21:07

github-actions bot added the Stale label Feb 8, 2023

atoulme changed the title ~~[receiver/cloudflarereceiver] New Alpha Logs Componenet cloudflarereceiver~~ [receiver/cloudflarereceiver] New Alpha Logs Component cloudflarereceiver Feb 12, 2023

atoulme reviewed Feb 12, 2023

View reviewed changes

github-actions bot removed the Stale label Feb 13, 2023

JonathanWamsley force-pushed the add-cloudflarereceiver branch 5 times, most recently from 94c5c97 to 89eb6cc Compare February 14, 2023 18:10

JonathanWamsley added 3 commits February 14, 2023 13:44

wip, testing client

a943f69

fix client w/ tests

22b3cfd

fix formatting/lint issues

b955160

JonathanWamsley added 11 commits February 14, 2023 13:44

add otel requirements

a2d8ce4

update go modules

c689d1c

addressing pr feedback

434bffa

update go mod

f8eee42

apply pr feedback

dd8141a

fix go mod

e373a99

remove race condition

ca7927c

removed mutating config required fields to a check validating require…

8fbbc8e

…d fields when custom fields are supplied

add generated github issue

ffa36da

run make commands genoteltestbedcol

da3bb6d

fix import order

abd7499

JonathanWamsley force-pushed the add-cloudflarereceiver branch from 89eb6cc to abd7499 Compare February 14, 2023 18:50

github-actions bot added the Stale label Mar 1, 2023

JonathanWamsley closed this Mar 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[receiver/cloudflarereceiver] New Alpha Logs Component cloudflarereceiver #17993

[receiver/cloudflarereceiver] New Alpha Logs Component cloudflarereceiver #17993

JonathanWamsley commented Jan 24, 2023 •

edited

Loading

runforesight bot commented Jan 24, 2023 •

edited

Loading

`build-and-test-windows` duration(4 seconds) has decreased 40 minutes 46 seconds compared to main branch avg(40 minutes 50 seconds).

✅ tracegen workflow has finished in 5 minutes 19 seconds (⚠️ 2 minutes 52 seconds more than `main` branch avg.) and finished at 24th Jan, 2023.

⭕ build-and-test-windows workflow has finished in 4 seconds (40 minutes 46 seconds less than `main` branch avg.) and finished at 14th Feb, 2023.

✅ telemetrygen workflow has finished in 54 seconds (2 minutes 12 seconds less than `main` branch avg.) and finished at 14th Feb, 2023.

✅ check-links workflow has finished in 1 minute 28 seconds (1 minute 6 seconds less than `main` branch avg.) and finished at 14th Feb, 2023.

✅ changelog workflow has finished in 2 minutes 19 seconds (1 minute 24 seconds less than `main` branch avg.) and finished at 14th Feb, 2023.

✅ prometheus-compliance-tests workflow has finished in 14 minutes 28 seconds (⚠️ 5 minutes 39 seconds more than `main` branch avg.) and finished at 14th Feb, 2023.

❌ build-and-test workflow has finished in 35 minutes 7 seconds (30 minutes 29 seconds less than `main` branch avg.) and finished at 14th Feb, 2023. 4 jobs failed. There are 2 test failures.

✅ load-tests workflow has finished in 18 minutes 38 seconds and finished at 14th Feb, 2023.

✅ e2e-tests workflow has finished in 17 minutes 1 second and finished at 14th Feb, 2023.

atoulme Jan 24, 2023

JonathanWamsley Jan 24, 2023

atoulme Jan 24, 2023

JonathanWamsley Jan 24, 2023

github-actions bot commented Feb 8, 2023

atoulme Feb 12, 2023

JonathanWamsley Feb 14, 2023

atoulme Feb 12, 2023

atoulme Feb 12, 2023

JonathanWamsley Feb 14, 2023

atoulme Feb 12, 2023

JonathanWamsley Feb 14, 2023

atoulme Feb 12, 2023

JonathanWamsley Feb 14, 2023

atoulme left a comment

github-actions bot commented Mar 1, 2023

JonathanWamsley commented Mar 3, 2023

	l.logger.Debug("starting cloudflare receiver in retrieval mode")
	l.logger.Debug("starting cloudflare receiver")

[receiver/cloudflarereceiver] New Alpha Logs Component cloudflarereceiver #17993

[receiver/cloudflarereceiver] New Alpha Logs Component cloudflarereceiver #17993

Conversation

JonathanWamsley commented Jan 24, 2023 • edited Loading

runforesight bot commented Jan 24, 2023 • edited Loading

Foresight Summary

build-and-test-windows duration(4 seconds) has decreased 40 minutes 46 seconds compared to main branch avg(40 minutes 50 seconds).

✅ tracegen workflow has finished in 5 minutes 19 seconds (⚠️ 2 minutes 52 seconds more than main branch avg.) and finished at 24th Jan, 2023.

⭕ build-and-test-windows workflow has finished in 4 seconds (40 minutes 46 seconds less than main branch avg.) and finished at 14th Feb, 2023.

✅ telemetrygen workflow has finished in 54 seconds (2 minutes 12 seconds less than main branch avg.) and finished at 14th Feb, 2023.

✅ check-links workflow has finished in 1 minute 28 seconds (1 minute 6 seconds less than main branch avg.) and finished at 14th Feb, 2023.

✅ changelog workflow has finished in 2 minutes 19 seconds (1 minute 24 seconds less than main branch avg.) and finished at 14th Feb, 2023.

✅ prometheus-compliance-tests workflow has finished in 14 minutes 28 seconds (⚠️ 5 minutes 39 seconds more than main branch avg.) and finished at 14th Feb, 2023.

❌ build-and-test workflow has finished in 35 minutes 7 seconds (30 minutes 29 seconds less than main branch avg.) and finished at 14th Feb, 2023. 4 jobs failed. There are 2 test failures.

✅ load-tests workflow has finished in 18 minutes 38 seconds and finished at 14th Feb, 2023.

✅ e2e-tests workflow has finished in 17 minutes 1 second and finished at 14th Feb, 2023.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Feb 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

atoulme left a comment

Choose a reason for hiding this comment

github-actions bot commented Mar 1, 2023

JonathanWamsley commented Mar 3, 2023

JonathanWamsley commented Jan 24, 2023 •

edited

Loading

runforesight bot commented Jan 24, 2023 •

edited

Loading

`build-and-test-windows` duration(4 seconds) has decreased 40 minutes 46 seconds compared to main branch avg(40 minutes 50 seconds).

✅ tracegen workflow has finished in 5 minutes 19 seconds (⚠️ 2 minutes 52 seconds more than `main` branch avg.) and finished at 24th Jan, 2023.

⭕ build-and-test-windows workflow has finished in 4 seconds (40 minutes 46 seconds less than `main` branch avg.) and finished at 14th Feb, 2023.

✅ telemetrygen workflow has finished in 54 seconds (2 minutes 12 seconds less than `main` branch avg.) and finished at 14th Feb, 2023.

✅ check-links workflow has finished in 1 minute 28 seconds (1 minute 6 seconds less than `main` branch avg.) and finished at 14th Feb, 2023.

✅ changelog workflow has finished in 2 minutes 19 seconds (1 minute 24 seconds less than `main` branch avg.) and finished at 14th Feb, 2023.

✅ prometheus-compliance-tests workflow has finished in 14 minutes 28 seconds (⚠️ 5 minutes 39 seconds more than `main` branch avg.) and finished at 14th Feb, 2023.

❌ build-and-test workflow has finished in 35 minutes 7 seconds (30 minutes 29 seconds less than `main` branch avg.) and finished at 14th Feb, 2023. 4 jobs failed. There are 2 test failures.