Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[receiver/awscloudwatch]: Use next token in Log Group discovery #32057

Merged
merged 15 commits into from
May 1, 2024

Conversation

schmikei
Copy link
Contributor

@schmikei schmikei commented Apr 1, 2024

Description:
Fixing a bug with the Log Group Discovery where nextToken was not properly getting set on the request.

Link to tracking Issue: Resolves #32053

Testing:

Documentation:

@schmikei schmikei marked this pull request as ready for review April 4, 2024 19:47
@schmikei schmikei requested a review from a team April 4, 2024 19:47
…tor-contrib into use-next-token-in-discovery
…tor-contrib into use-next-token-in-discovery
@AllanOricil
Copy link

AllanOricil commented Apr 11, 2024

@schmikei it does not work

I used the same config.yaml with the binary I created from your branch and both had different results.

Below is the output when using 0.97.0

2024-04-11T04:59:12.112Z	info	prometheusreceiver@v0.97.0/metrics_receiver.go:299	Starting scrape manager	{"kind": "receiver", "name": "prometheus", "data_type": "metrics"}
2024-04-11T04:59:21.971Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:287	attempting to discover log groups.	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100}
2024-04-11T04:59:22.003Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n  CreationTime: 1701809123724,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/lambda/get-instances-api-function\",\n  MetricFilterCount: 0,\n  RetentionInDays: 7,\n  StoredBytes: 137470\n}"}
2024-04-11T04:59:31.970Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:287	attempting to discover log groups.	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100}
2024-04-11T04:59:31.994Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n  CreationTime: 1701809123724,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/lambda/get-instances-api-function\",\n  MetricFilterCount: 0,\n  RetentionInDays: 7,\n  StoredBytes: 137470\n}"}
2024-04-11T04:59:41.971Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:287	attempting to discover log groups.	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100}
2024-04-11T04:59:41.997Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:323	discovered log group	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n  Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n  CreationTime: 1701809123724,\n  LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n  LogGroupClass: \"STANDARD\",\n  LogGroupName: \"/aws/lambda/get-instances-api-function\",\n  MetricFilterCount: 0,\n  RetentionInDays: 7,\n  StoredBytes: 137470\n}"}

Below you can see the result I got from the new binary

2024-04-11T04:58:36.200Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:287	attempting to discover log groups.	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100}
2024-04-11T04:58:36.200Z	error	awscloudwatchreceiver@v0.97.0/logs.go:165	unable to perform discovery of log groups	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "error": "unable to list log groups: InvalidParameter: 1 validation error(s) found.\n- minimum field size of 1, DescribeLogGroupsInput.NextToken.\n"}
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awscloudwatchreceiver.(*logsReceiver).startPolling
	github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awscloudwatchreceiver@v0.97.0/logs.go:165
2024-04-11T04:58:46.201Z	debug	awscloudwatchreceiver@v0.97.0/logs.go:287	attempting to discover log groups.	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100}
2024-04-11T04:58:46.201Z	error	awscloudwatchreceiver@v0.97.0/logs.go:165	unable to perform discovery of log groups	{"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "error": "unable to list log groups: InvalidParameter: 1 validation error(s) found.\n- minimum field size of 1, DescribeLogGroupsInput.NextToken.\n"}
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awscloudwatchreceiver.(*logsReceiver).startPolling

@AllanOricil
Copy link

How can I test changes without having to build the binary? it takes too much time

Copy link
Contributor

This PR was marked stale due to lack of activity. It will be closed in 14 days.

@github-actions github-actions bot added the Stale label Apr 30, 2024
@AllanOricil
Copy link

😭

@github-actions github-actions bot removed the Stale label May 1, 2024
@djaglowski djaglowski merged commit 088d272 into open-telemetry:main May 1, 2024
170 checks passed
@github-actions github-actions bot added this to the next release milestone May 1, 2024
@schmikei schmikei deleted the use-next-token-in-discovery branch June 26, 2024 18:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] AWS cloudwatch receiver doesn't use next token while discovering log groups
4 participants