[receiver/awscloudwatch] Missing log stream events #32231
Comments
AllanOricil
added
bug
|
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
|
It is also not iterating over all log groups I have in my aws account. It iterates over and over again in log groups that dont even have log streams anymore, and it also never tries other log groups than these ones shown below. 
|
|
@djaglowski @schmikei could you guys help me to get my dev env so that I can fix this? |
|
is this #32057 the fix? |
|
I think it might be related, if you'd be willing to try out the PR feel free to see if it resolves your issue, we'd very much appreciate it! |
|
@schmikei how can I do it? Im using the otel-collector binary |
You can checkout the fork and do a |
|
@schmikei can I build linux dis on a mac? |
|
never mind @schmikei |
|
@schmikei do you have this error while building? I checkout your branch, and then I run the |
|
@schmikei I spent the day building this stuff but it did not work :/ I used the same Below is the output when using 0.97.0 Below you can see the result I got from the new binary |
|
@djaglowski @schmikei can you help here? |
|
@AllanOricil did you end up trying the latest release? I've validated that everything with the receiver is currently working as expected in my lab and I'm currently unable to replicate your specific behavior at the moment. If you want to start narrowing down your config to target specific like in our examples: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/awscloudwatchreceiver#sample-configs Apologies that I cannot be more helpful, if you do find something, always happy to review a PR 😄 |
|
I have a similar issue. My receiver config: Thanks |
|
Aah, i see this: |
|
Ok, so i did some tests with AWS API Gateway. Of those 1000 requests 1000 are processed by AWS API Gateway. I add here the timestamps of the Cloudwatch logs and the loki logs. When you compare the two file you see about every minute (my poll interval) a gap in the logs. |
|
Increasing or decreasing the poll interval does not help. |
|
I'm starting to think Cloudwatch may take a second to serve the log entries on the API (just a suspicion), created a Spike PR in #33809 that will set the next end time... I will try to dedicate some time to see if I can replicate the specific behavior, but if you'd like to test out the PR to see if it helps, it would help as I try and replicate your test case. |
|
@schmikei once I have time I will test it again |
|
I did an other test this time with only API Gateway log groups. That is 4 log groups out of the 38 that i have in total. The gaps in the logs that i see are between 14-18 seconds with a 1min pull request. @schmikei I don't have the mad skills to do this. But if you can provide a docker, i can test. |
|
@schmikei Here is a graph of the logs i get. This is done with a poll of 5 min and you see a big gap every 5 min. When you look closely you also see small gaps every couple of seconds. It turns out that when Cloudwatch created a new log stream the first logs (API Gateway call) is not picked up. A possible solution to fix this would be to work with a time offset equal to the poll rate, so if |
|
@AllanOricil @schmikei |
|
I did not have time to test the latest changes. If you did and the issue is still there, make a video and post it here. |
|
Heya @AllanOricil |
|
@schmikei |
|
@Jessimon I cant invest time on testing it again. Once I reach a point in my project where It makes sense to add probes to every single running service, I will try again. |
Component(s)
receiver/awscloudwatch
What happened?
Description
1 - Past log streams aren't exported
2 - Log streams are incomplete
Steps to Reproduce
1 - create a lambda function that logs more than 15 log lines in Cloud Watch
2 - run this function several times and ensure a single log stream has more than 15 log lines
3 - in an EC2 machine, run otel-collector with this receiver
WARNING: don't forget to change
NAME_OF_YOUR_LAMBDA_FUNCTION_LOG_GROUPby the name of the log group of your lambda functionDon't forget to register the receiver in the logs pipeline
4 - verify that logs are processed with no error. You should see something as shown below
Expected Result
Actual Result
Collector version
0.97.0
Environment information
Environment
NAME="Amazon Linux"
VERSION="2023"
ID="amzn"
ID_LIKE="fedora"
VERSION_ID="2023"
PLATFORM_ID="platform:al2023"
PRETTY_NAME="Amazon Linux 2023"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2023"
HOME_URL="https://aws.amazon.com/linux/"
BUG_REPORT_URL="https://github.com/amazonlinux/amazon-linux-2023"
SUPPORT_END="2028-03-15"
OpenTelemetry Collector configuration
Log output
2024-04-09T01:32:28.424Z debug awscloudwatchreceiver@v0.97.0/logs.go:323 discovered log group {"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n CreationTime: 1701809123724,\n LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n LogGroupClass: \"STANDARD\",\n LogGroupName: \"/aws/lambda/get-instances-api-function\",\n MetricFilterCount: 0,\n RetentionInDays: 7,\n StoredBytes: 113613\n}"} 2024-04-09T01:32:38.361Z debug awscloudwatchreceiver@v0.97.0/logs.go:287 attempting to discover log groups. {"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100} 2024-04-09T01:32:38.386Z debug awscloudwatchreceiver@v0.97.0/logs.go:323 discovered log group {"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n CreationTime: 1701809123724,\n LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n LogGroupClass: \"STANDARD\",\n LogGroupName: \"/aws/lambda/get-instances-api-function\",\n MetricFilterCount: 0,\n RetentionInDays: 7,\n StoredBytes: 113613\n}"} 2024-04-09T01:32:48.362Z debug awscloudwatchreceiver@v0.97.0/logs.go:287 attempting to discover log groups. {"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100} 2024-04-09T01:32:48.382Z debug awscloudwatchreceiver@v0.97.0/logs.go:323 discovered log group {"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n CreationTime: 1701809123724,\n LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n LogGroupClass: \"STANDARD\",\n LogGroupName: \"/aws/lambda/get-instances-api-function\",\n MetricFilterCount: 0,\n RetentionInDays: 7,\n StoredBytes: 113613\n}"} 2024-04-09T01:32:48.468Z info LogsExporter {"kind": "exporter", "data_type": "logs", "name": "debug", "resource logs": 1, "log records": 15} 2024-04-09T01:32:48.469Z info ResourceLog #0 Resource SchemaURL: Resource attributes: -> aws.region: Str(us-east-2) -> cloudwatch.log.group.name: Str(/aws/lambda/get-instances-api-function) -> cloudwatch.log.stream: Str(2024/04/09/[$LATEST]9a79cb34998a4037bf1e3ff5df35fea5) ScopeLogs #0 ScopeLogs SchemaURL: InstrumentationScope LogRecord #0 ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC Timestamp: 2024-04-09 01:32:41.354 +0000 UTC SeverityText: SeverityNumber: Unspecified(0) Body: Str(INIT_START Runtime Version: nodejs:18.v26 Runtime Version ARN: arn:aws:lambda:us-east-2::runtime:0cdcfbdefbc5e7d3343f73c2e2dd3cba17d61dea0686b404502a0c9ce83931b9 ) Attributes: -> id: Str(38192844104842187460300983962019760926010078374443876352) Trace ID: Span ID: Flags: 0 LogRecord #1 ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC Timestamp: 2024-04-09 01:32:42.498 +0000 UTC SeverityText: SeverityNumber: Unspecified(0) Body: Str(START RequestId: 9f7b878f-a62a-47d1-be33-9526adf64f30 Version: $LATEST ) Attributes: -> id: Str(38192844130354239967420016835936622629919803937285472257) Trace ID: Span ID: Flags: 0 LogRecord #2 ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC Timestamp: 2024-04-09 01:32:42.571 +0000 UTC SeverityText: SeverityNumber: Unspecified(0) Body: Str({"body":null,"headers":{"Accept":"application/json, text/plain, */*","Accept-Encoding":"gzip, deflate, br, zstd","Accept-Language":"en-GB,en-US;q=0.9,en;q=0.8,pt;q=0.7","Authorization":"eyJraWQiOiJvSUdwUUVhaFIxOWlMc05GbWhpVHpsRkNYUEx3eDcxNDN0S3hGaEFUNEJBPSIsImFsZyI6IlJTMjU2In0.eyJhdF9oYXNoIjoiUTZHdUZyLXAtREJvd3ZacElwZW81QSIsInN1YiI6IjgxNGIxNTYwLWIwNTEtNzA0MC1iNThiLWIyMThiNDJjZTU5YiIsImN1c3RvbTpzdWJkb21haW4iOiJhbGxhbm9yaWNpbCIsImNvZ25pdG86Z3JvdXBzIjpbInVzLWVhc3QtMl84QmVYSm42M2RfR29vZ2xlIiwiZDFkNDI5NTAtZGFiYi00M2VmLWI3Y2ItMDY3ODRiOGNmNDViIl0sImlzcyI6Imh0dHBzOlwvXC9jb2duaXRvLWlkcC51cy1lYXN0LTIuYW1hem9uYXdzLmNvbVwvdXMtZWFzdC0yXzhCZVhKbjYzZCIsImlkZW50aXRpZXMiOlt7ImRhdGVDcmVhdGVkIjoiMTcxMTExNjcwOTcwNyIsInVzZXJJZCI6IjEwNDExNjkxMTE3NDI0NjE1NzU5NSIsInByb3ZpZGVyTmFtZSI6Ikdvb2dsZSIsInByb3ZpZGVyVHlwZSI6Ikdvb2dsZSIsImlzc3VlciI6bnVsbCwicHJpbWFyeSI6InRydWUifV0sImF1dGhfdGltZSI6MTcxMjQzNTE1MywiZXhwIjoxNzEyNzEyNzYwLCJpYXQiOjE3MTI2MjYzNjAsImp0aSI6ImZjNWE3NDkzLTFjOWQtNGUyMC1hMGViLThjYzMzYzlmMTdmMiIsImVtYWlsIjoiYWxsbGFub3JpY2lsQGdtYWlsLmNvbSIsImN1c3RvbTpwc19zdWJzY3JpcHRpb25faWQiOiJzdWJfMU94OHFWSFl3TUVKQjQ2OFFGWlNMeThHIiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJjdXN0b206cHNfY3VzdG9tZXJfaWQiOiJjdXNfUG1pSFowanV4RnAzTHYiLCJjdXN0b206b3JnYW5pemF0aW9uX2lkIjoiZDFkNDI5NTAtZGFiYi00M2VmLWI3Y2ItMDY3ODRiOGNmNDViIiwiY3VzdG9tOnBzX3JlZGlyZWN0X3VybCI6Imh0dHA6XC9cL25vZGUtcmVhZHkuY29tXC9pbnN0YW5jZS1wcm92aXNpb25pbmctam9ic1wvY2FsbGJhY2siLCJjdXN0b206b3JnYW5pemF0aW9uX25hbWUiOiJBbGxhbiBPcmljaWwiLCJjb2duaXRvOnVzZXJuYW1lIjoiR29vZ2xlXzEwNDExNjkxMTE3NDI0NjE1NzU5NSIsImN1c3RvbTpzZXR1cCI6InRydWUiLCJnaXZlbl9uYW1lIjoiQWxsYW4gR29uw6dhbHZlcyBHb21lcyIsIm9yaWdpbl9qdGkiOiIwMzI5NTkwOC1hYjIzLTQ2NzAtODVjNS1mOTQ3NjhiNmQ3OGYiLCJhdWQiOiIxazMxY2Q4NzRoMTRkNHU4aTcyNGlkNDJwbiIsImN1c3RvbTpjcmVhdGVfaW5zdGFuY2UiOiJ0cnVlIiwidG9rZW5fdXNlIjoiaWQiLCJmYW1pbHlfbmFtZSI6Ik9yaWNpbCJ9.Srb-vdOkRbkNcq0ZvHUDYT1cl9GJ-F_XM8est0yzehoOKsfD72e4_qhWcwCRGCp1ZRd9KmFxp7rhQiqzTjccXjNNSORl823c8fM3sDLSpUF6-llLvdf9zDaH5JZxOwMX-wIfJmtRCgWjYmCEJpZatPHG2M3e8jVvAFzB-EAaEs9AfR6Re5YhVtzy5aUcHUxkMw_mCtrEmg2V-zY2SEyIg5m6EMQ6J5AfKsDe3nteM28Sa0IoWOxGf5KhBySUC49RawQCfO0vYNYXC3PVJNfmTaKJ8P2g4kXtCpi11RS7ONMth2ynrPg-eWTiI_I6g9pQdvsAruomontiYrvH4kknNw","CloudFront-Forwarded-Proto":"https","CloudFront-Is-Desktop-Viewer":"true","CloudFront-Is-Mobile-Viewer":"false","CloudFront-Is-SmartTV-Viewer":"false","CloudFront-Is-Tablet-Viewer":"false","CloudFront-Viewer-ASN":"269073","CloudFront-Viewer-Country":"BR","Host":"vjydkvkskl.execute-api.us-east-2.amazonaws.com","Referer":"https://node-ready.com/","User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36","Via":"2.0 6fe8e2d5db6a80353eb675f61c249810.cloudfront.net (CloudFront)","X-Amz-Cf-Id":"kgg1BBFHKR4861YdGx-Kcf1-149AGIp-EUSUm9_J4MFT5WvC0cH3PA==","X-Amzn-Trace-Id":"Root=1-66149ab8-0a47130e4ff50bbf715025c0","X-Forwarded-For":"138.94.127.146, 15.158.19.244","X-Forwarded-Port":"443","X-Forwarded-Proto":"https","origin":"https://node-ready.com","sec-ch-ua":"\"Brave\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"macOS\"","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"cross-site","sec-gpc":"1"},"httpMethod":"GET","isBase64Encoded":false,"level":"info","message":"event","multiValueHeaders":{"Accept":["application/json, text/plain, */*"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Accept-Language":["en-GB,en-US;q=0.9,en;q=0.8,pt;q=0.7"],"Authorization":["eyJraWQiOiJvSUdwUUVhaFIxOWlMc05GbWhpVHpsRkNYUEx3eDcxNDN0S3hGaEFUNEJBPSIsImFsZyI6IlJTMjU2In0.eyJhdF9oYXNoIjoiUTZHdUZyLXAtREJvd3ZacElwZW81QSIsInN1YiI6IjgxNGIxNTYwLWIwNTEtNzA0MC1iNThiLWIyMThiNDJjZTU5YiIsImN1c3RvbTpzdWJkb21haW4iOiJhbGxhbm9yaWNpbCIsImNvZ25pdG86Z3JvdXBzIjpbInVzLWVhc3QtMl84QmVYSm42M2RfR29vZ2xlIiwiZDFkNDI5NTAtZGFiYi00M2VmLWI3Y2ItMDY3ODRiOGNmNDViIl0sImlzcyI6Imh0dHBzOlwvXC9jb2duaXRvLWlkcC51cy1lYXN0LTIuYW1hem9uYXdzLmNvbVwvdXMtZWFzdC0yXzhCZVhKbjYzZCIsImlkZW50aXRpZXMiOlt7ImRhdGVDcmVhdGVkIjoiMTcxMTExNjcwOTcwNyIsInVzZXJJZCI6IjEwNDExNjkxMTE3NDI0NjE1NzU5NSIsInByb3ZpZGVyTmFtZSI6Ikdvb2dsZSIsInByb3ZpZGVyVHlwZSI6Ikdvb2dsZSIsImlzc3VlciI6bnVsbCwicHJpbWFyeSI6InRydWUifV0sImF1dGhfdGltZSI6MTcxMjQzNTE1MywiZXhwIjoxNzEyNzEyNzYwLCJpYXQiOjE3MTI2MjYzNjAsImp0aSI6ImZjNWE3NDkzLTFjOWQtNGUyMC1hMGViLThjYzMzYzlmMTdmMiIsImVtYWlsIjoiYWxsbGFub3JpY2lsQGdtYWlsLmNvbSIsImN1c3RvbTpwc19zdWJzY3JpcHRpb25faWQiOiJzdWJfMU94OHFWSFl3TUVKQjQ2OFFGWlNMeThHIiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJjdXN0b206cHNfY3VzdG9tZXJfaWQiOiJjdXNfUG1pSFowanV4RnAzTHYiLCJjdXN0b206b3JnYW5pemF0aW9uX2lkIjoiZDFkNDI5NTAtZGFiYi00M2VmLWI3Y2ItMDY3ODRiOGNmNDViIiwiY3VzdG9tOnBzX3JlZGlyZWN0X3VybCI6Imh0dHA6XC9cL25vZGUtcmVhZHkuY29tXC9pbnN0YW5jZS1wcm92aXNpb25pbmctam9ic1wvY2FsbGJhY2siLCJjdXN0b206b3JnYW5pemF0aW9uX25hbWUiOiJBbGxhbiBPcmljaWwiLCJjb2duaXRvOnVzZXJuYW1lIjoiR29vZ2xlXzEwNDExNjkxMTE3NDI0NjE1NzU5NSIsImN1c3RvbTpzZXR1cCI6InRydWUiLCJnaXZlbl9uYW1lIjoiQWxsYW4gR29uw6dhbHZlcyBHb21lcyIsIm9yaWdpbl9qdGkiOiIwMzI5NTkwOC1hYjIzLTQ2NzAtODVjNS1mOTQ3NjhiNmQ3OGYiLCJhdWQiOiIxazMxY2Q4NzRoMTRkNHU4aTcyNGlkNDJwbiIsImN1c3RvbTpjcmVhdGVfaW5zdGFuY2UiOiJ0cnVlIiwidG9rZW5fdXNlIjoiaWQiLCJmYW1pbHlfbmFtZSI6Ik9yaWNpbCJ9.Srb-vdOkRbkNcq0ZvHUDYT1cl9GJ-F_XM8est0yzehoOKsfD72e4_qhWcwCRGCp1ZRd9KmFxp7rhQiqzTjccXjNNSORl823c8fM3sDLSpUF6-llLvdf9zDaH5JZxOwMX-wIfJmtRCgWjYmCEJpZatPHG2M3e8jVvAFzB-EAaEs9AfR6Re5YhVtzy5aUcHUxkMw_mCtrEmg2V-zY2SEyIg5m6EMQ6J5AfKsDe3nteM28Sa0IoWOxGf5KhBySUC49RawQCfO0vYNYXC3PVJNfmTaKJ8P2g4kXtCpi11RS7ONMth2ynrPg-eWTiI_I6g9pQdvsAruomontiYrvH4kknNw"],"CloudFront-Forwarded-Proto":["https"],"CloudFront-Is-Desktop-Viewer":["true"],"CloudFront-Is-Mobile-Viewer":["false"],"CloudFront-Is-SmartTV-Viewer":["false"],"CloudFront-Is-Tablet-Viewer":["false"],"CloudFront-Viewer-ASN":["269073"],"CloudFront-Viewer-Country":["BR"],"Host":["vjydkvkskl.execute-api.us-east-2.amazonaws.com"],"Referer":["https://node-ready.com/"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"],"Via":["2.0 6fe8e2d5db6a80353eb675f61c249810.cloudfront.net (CloudFront)"],"X-Amz-Cf-Id":["kgg1BBFHKR4861YdGx-Kcf1-149AGIp-EUSUm9_J4MFT5WvC0cH3PA=="],"X-Amzn-Trace-Id":["Root=1-66149ab8-0a47130e4ff50bbf715025c0"],"X-Forwarded-For":["138.94.127.146, 15.158.19.244"],"X-Forwarded-Port":["443"],"X-Forwarded-Proto":["https"],"origin":["https://node-ready.com"],"sec-ch-ua":["\"Brave\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\""],"sec-ch-ua-mobile":["?0"],"sec-ch-ua-platform":["\"macOS\""],"sec-fetch-dest":["empty"],"sec-fetch-mode":["cors"],"sec-fetch-site":["cross-site"],"sec-gpc":["1"]},"multiValueQueryStringParameters":null,"path":"/instances","pathParameters":null,"queryStringParameters":null,"requestContext":{"accountId":"845044614340","apiId":"vjydkvkskl","authorizer":{"claims":{"at_hash":"Q6GuFr-p-DBowvZpIpeo5A","aud":"1k31cd874h14d4u8i724id42pn","auth_time":"1712435153","cognito:groups":"us-east-2_8BeXJn63d_Google,d1d42950-dabb-43ef-b7cb-06784b8cf45b","cognito:username":"Google_104116911174246157595","custom:create_instance":"true","custom:organization_id":"d1d42950-dabb-43ef-b7cb-06784b8cf45b","custom:organization_name":"Allan Oricil","custom:ps_customer_id":"cus_PmiHZ0juxFp3Lv","custom:ps_redirect_url":"http://node-ready.com/instance-provisioning-jobs/callback","custom:ps_subscription_id":"sub_1Ox8qVHYwMEJB468QFZSLy8G","custom:setup":"true","custom:subdomain":"allanoricil","email":"alllanoricil@gmail.com","email_verified":"false","exp":"Wed Apr 10 01:32:40 UTC 2024","family_name":"Oricil","given_name":"Allan Gonçalves Gomes","iat":"Tue Apr 09 01:32:40 UTC 2024","identities":"{\"dateCreated\":\"1711116709707\",\"userId\":\"104116911174246157595\",\"providerName\":\"Google\",\"providerType\":\"Google\",\"issuer\":null,\"primary\":\"true\"}","iss":"https://cognito-idp.us-east-2.amazonaws.com/us-east-2_8BeXJn63d","jti":"fc5a7493-1c9d-4e20-a0eb-8cc33c9f17f2","origin_jti":"03295908-ab23-4670-85c5-f94768b6d78f","sub":"814b1560-b051-7040-b58b-b218b42ce59b","token_use":"id"}},"deploymentId":"ube8ur","domainName":"vjydkvkskl.execute-api.us-east-2.amazonaws.com","domainPrefix":"vjydkvkskl","extendedRequestId":"V70c7FCmiYcED6Q=","httpMethod":"GET","identity":{"accessKey":null,"accountId":null,"caller":null,"cognitoAuthenticationProvider":null,"cognitoAuthenticationType":null,"cognitoIdentityId":null,"cognitoIdentityPoolId":null,"principalOrgId":null,"sourceIp":"138.94.127.146","user":null,"userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36","userArn":null},"operationName":"This method can be used to list all instances from an account","path":"/v1/instances","protocol":"HTTP/1.1","requestId":"4b2c442e-a405-4659-8e62-6a58bb578993","requestTime":"09/Apr/2024:01:32:40 +0000","requestTimeEpoch":1712626360898,"resourceId":"t4baq8","resourcePath":"/instances","stage":"v1"},"requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","resource":"/instances","stageVariables":null,"timestamp":"2024-04-09T01:32:42.513Z"} ) Attributes: -> id: Str(38192844131982194366912752325268730063823134327222042626) Trace ID: Span ID: Flags: 0 LogRecord #3 ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC Timestamp: 2024-04-09 01:32:42.572 +0000 UTC SeverityText: SeverityNumber: Unspecified(0) Body: Str({"level":"debug","message":"------------- headers -------------","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:42.572Z"} ) Attributes: -> id: Str(38192844132004495112111282948410265782095782688728023043) Trace ID: Span ID: Flags: 0 LogRecord #4 ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC Timestamp: 2024-04-09 01:32:42.572 +0000 UTC SeverityText: SeverityNumber: Unspecified(0) Body: Str({"level":"debug","message":"origin","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:42.572Z"} ) Attributes: -> id: Str(38192844132004495112111282948410265782095782688728023044) Trace ID: Span ID: Flags: 0 LogRecord #5 ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC Timestamp: 2024-04-09 01:32:42.573 +0000 UTC SeverityText: SeverityNumber: Unspecified(0) Body: Str({"0":"http://localhost:8080","1":"https://node-ready.com","2":"https://instance-provisioning-admin-api-doc-bucket.s3.us-east-2.amazonaws.com","3":"https://instance-provisioning-api-doc-bucket.s3.us-east-2.amazonaws.com","level":"debug","message":"allowedOrigins","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:42.572Z"} ) Attributes: -> id: Str(38192844132026795857309813571551801500368431050234003461) Trace ID: Span ID: Flags: 0 LogRecord #6 ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC Timestamp: 2024-04-09 01:32:42.573 +0000 UTC SeverityText: SeverityNumber: Unspecified(0) Body: Str({"level":"debug","message":{"Access-Control-Allow-Credentials":true,"Access-Control-Allow-Headers":"Content-Type,Authorization,X-Amz-Date,X-Api-Key,X-Amz-Security-Token,X-Amz-User-Agent","Access-Control-Allow-Methods":"OPTIONS,GET","Access-Control-Allow-Origin":"https://node-ready.com","Content-Security-Policy":"default-src 'none'; connect-src 'self'; img-src 'none'; script-src 'self'; style-src 'none'; object-src 'none'; frame-src 'none'","Content-Type":"application/json","Cross-Origin-Embedder-Policy":"require-corp","Cross-Origin-Opener-Policy":"same-origin","Referrer-Policy":"no-referrer","Strict-Transport-Security":"max-age=63072000; includeSubdomains; preload","X-Content-Type-Options":"nosniff","X-Frame-Options":"DENY","X-Permitted-Cross-Domain-Policies":"none","X-XSS-Protection":"1; mode=block"},"requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:42.573Z"} ) Attributes: -> id: Str(38192844132026795857309813571551801500368431050234003462) Trace ID: Span ID: Flags: 0 LogRecord #7 ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC Timestamp: 2024-04-09 01:32:42.573 +0000 UTC SeverityText: SeverityNumber: Unspecified(0) Body: Str({"level":"debug","message":"------------- headers -------------","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:42.573Z"} ) Attributes: -> id: Str(38192844132026795857309813571551801500368431050234003463) Trace ID: Span ID: Flags: 0 LogRecord #8 ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC Timestamp: 2024-04-09 01:32:42.573 +0000 UTC SeverityText: SeverityNumber: Unspecified(0) Body: Str({"level":"info","message":"fetching secrets from secrets manager","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:42.573Z"} ) Attributes: -> id: Str(38192844132026795857309813571551801500368431050234003464) Trace ID: Span ID: Flags: 0 LogRecord #9 ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC Timestamp: 2024-04-09 01:32:43.433 +0000 UTC SeverityText: SeverityNumber: Unspecified(0) Body: Str({"level":"info","message":"*****","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:43.433Z"} ) Attributes: -> id: Str(38192844151205436728046149473272519214846021945377161225) Trace ID: Span ID: Flags: 0 LogRecord #10 ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC Timestamp: 2024-04-09 01:32:43.433 +0000 UTC SeverityText: SeverityNumber: Unspecified(0) Body: Str({"level":"debug","message":"creating new prisma client","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:43.433Z"} ) Attributes: -> id: Str(38192844151205436728046149473272519214846021945377161226) Trace ID: Span ID: Flags: 0 LogRecord #11 ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC Timestamp: 2024-04-09 01:32:43.434 +0000 UTC SeverityText: SeverityNumber: Unspecified(0) Body: Str({"level":"info","message":"*****","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:43.433Z"} ) Attributes: -> id: Str(38192844151227737473244680096414054933118670306883141643) Trace ID: Span ID: Flags: 0 LogRecord #12 ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC Timestamp: 2024-04-09 01:32:43.434 +0000 UTC SeverityText: SeverityNumber: Unspecified(0) Body: Str({"level":"debug","message":"","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:43.434Z"} ) Attributes: -> id: Str(38192844151227737473244680096414054933118670306883141644) Trace ID: Span ID: Flags: 0 LogRecord #13 ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC Timestamp: 2024-04-09 01:32:43.594 +0000 UTC SeverityText: SeverityNumber: Unspecified(0) Body: Str({"level":"info","message":"setting up sentry","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:43.591Z"} ) Attributes: -> id: Str(38192844154795856705009579799059769856742408147840008205) Trace ID: Span ID: Flags: 0 LogRecord #14 ObservedTimestamp: 2024-04-09 01:32:48.468672478 +0000 UTC Timestamp: 2024-04-09 01:32:43.752 +0000 UTC SeverityText: SeverityNumber: Unspecified(0) Body: Str({"level":"info","message":"finding all instances for: 814b1560-b051-7040-b58b-b218b42ce59b","requestId":"9f7b878f-a62a-47d1-be33-9526adf64f30","timestamp":"2024-04-09T01:32:43.752Z"} ) Attributes: -> id: Str(38192844158319374446377418255422413343820849265784913934) Trace ID: Span ID: Flags: 0 {"kind": "exporter", "data_type": "logs", "name": "debug"} 2024-04-09T01:32:58.361Z debug awscloudwatchreceiver@v0.97.0/logs.go:287 attempting to discover log groups. {"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100} 2024-04-09T01:32:58.381Z debug awscloudwatchreceiver@v0.97.0/logs.go:323 discovered log group {"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n CreationTime: 1701809123724,\n LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n LogGroupClass: \"STANDARD\",\n LogGroupName: \"/aws/lambda/get-instances-api-function\",\n MetricFilterCount: 0,\n RetentionInDays: 7,\n StoredBytes: 113613\n}"} 2024-04-09T01:33:08.362Z debug awscloudwatchreceiver@v0.97.0/logs.go:287 attempting to discover log groups. {"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100} 2024-04-09T01:33:08.382Z debug awscloudwatchreceiver@v0.97.0/logs.go:323 discovered log group {"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n CreationTime: 1701809123724,\n LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n LogGroupClass: \"STANDARD\",\n LogGroupName: \"/aws/lambda/get-instances-api-function\",\n MetricFilterCount: 0,\n RetentionInDays: 7,\n StoredBytes: 113613\n}"} 2024-04-09T01:33:18.362Z debug awscloudwatchreceiver@v0.97.0/logs.go:287 attempting to discover log groups. {"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100} 2024-04-09T01:33:18.380Z debug awscloudwatchreceiver@v0.97.0/logs.go:323 discovered log group {"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n CreationTime: 1701809123724,\n LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n LogGroupClass: \"STANDARD\",\n LogGroupName: \"/aws/lambda/get-instances-api-function\",\n MetricFilterCount: 0,\n RetentionInDays: 7,\n StoredBytes: 113613\n}"} 2024-04-09T01:33:28.362Z debug awscloudwatchreceiver@v0.97.0/logs.go:287 attempting to discover log groups. {"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100} 2024-04-09T01:33:28.380Z debug awscloudwatchreceiver@v0.97.0/logs.go:323 discovered log group {"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n CreationTime: 1701809123724,\n LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n LogGroupClass: \"STANDARD\",\n LogGroupName: \"/aws/lambda/get-instances-api-function\",\n MetricFilterCount: 0,\n RetentionInDays: 7,\n StoredBytes: 113613\n}"} 2024-04-09T01:33:38.362Z debug awscloudwatchreceiver@v0.97.0/logs.go:287 attempting to discover log groups. {"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100} 2024-04-09T01:33:38.379Z debug awscloudwatchreceiver@v0.97.0/logs.go:323 discovered log group {"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n CreationTime: 1701809123724,\n LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n LogGroupClass: \"STANDARD\",\n LogGroupName: \"/aws/lambda/get-instances-api-function\",\n MetricFilterCount: 0,\n RetentionInDays: 7,\n StoredBytes: 113613\n}"} 2024-04-09T01:33:48.361Z debug awscloudwatchreceiver@v0.97.0/logs.go:287 attempting to discover log groups. {"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100} 2024-04-09T01:33:48.382Z debug awscloudwatchreceiver@v0.97.0/logs.go:323 discovered log group {"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n CreationTime: 1701809123724,\n LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n LogGroupClass: \"STANDARD\",\n LogGroupName: \"/aws/lambda/get-instances-api-function\",\n MetricFilterCount: 0,\n RetentionInDays: 7,\n StoredBytes: 113613\n}"} 2024-04-09T01:33:58.362Z debug awscloudwatchreceiver@v0.97.0/logs.go:287 attempting to discover log groups. {"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 100} 2024-04-09T01:33:58.385Z debug awscloudwatchreceiver@v0.97.0/logs.go:323 discovered log group {"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n Arn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function:*\",\n CreationTime: 1701809123724,\n LogGroupArn: \"arn:aws:logs:us-east-2:845044614340:log-group:/aws/lambda/get-instances-api-function\",\n LogGroupClass: \"STANDARD\",\n LogGroupName: \"/aws/lambda/get-instances-api-function\",\n MetricFilterCount: 0,\n RetentionInDays: 7,\n StoredBytes: 113613\n}"}Additional context
Im using Signoz to see my logs
Compare the images to verify that there are missing log events for my log stream named
2024/04/09/[$LATEST]9a79cb34998a4037bf1e3ff5df35fea5In Signoz, I filtered logs by the log stream name
The text was updated successfully, but these errors were encountered: