Error while Creating a stack - Kinesis.AccessDenied

[anithac76]

Hello,

Greetings. My name is Anitha and I am a newbie to AWS. I came across your Cloudwatch2S3 blog and was super excited to try it out. I followed the instructions and Created a stack by uploading your Cloud Formation Template.
Stack Creation did not complete successfully and got rolled back. I did notice an error in the following Log Group / Log Stream:
I am unable to proceed, immensely appreciate if you can help me.

Log Groups/aws/kinesisfirehose/CloudWatch2S3-DeliveryStreamS3Delivery

{
"deliveryStreamARN": "arn:aws:firehose:us-east-1:017396793107:deliverystream/CloudWatch2S3-DeliveryStream-7K2JT6X19VY9",
"destination": "arn:aws:s3:::cloudwatch2s3-logbucket-y6bdn4u1akzc",
"deliveryStreamVersionId": 1,
"message": "Access was denied when calling Kinesis. Ensure the access policy on the IAM role used allows access to the appropriate Kinesis APIs.",
"errorCode": "Kinesis.AccessDenied",
"processor": "arn:aws:lambda:us-east-1:017396793107:function:CloudWatch2S3-LogProcessorFunction-BV2QL3V5RVA8"
}

I login to my AWS account as a root user and i have the following policy attached. (Screenshot attached)

Thanks,
Anitha

[kichik]

That error shows up for a successful deployment and can be ignored. What is the error in CloudFormation?

[kichik]

I had to delete your comment as it had your credentials in it. This is a public page. Anyone can see this. Please re-post without that.

[kichik]

Please re-post.

[anithac76]

Thanks for your response.

1. Cloud Formation Stack showed "In-progress" status for over 1 hour. I did not see any errors in CF Stack Event or Resources Tab. It created all the resources except the Subscriber for over an hour. Screen Shot below. After 1 hour it Rolled back all the resources created.

2. Where can I locate the CF Stack Creation errors - to check what went wrong during Stack Creation and why it got Rolled back.

3. For now, I just have one account, so i set the AllowedAccounts parameter as 01739679XXXX (my AWS Account # ). Is it right ? I have a ec2 server running (demo-webserver) and want to capture webserver logs sent to the CloudWatch Log group demoWebServer.

4. I was able to check all the resources created before they were all rolled back after an hour since I created the CF stack, everything appeared fine.
   Except that none of the raw data Log Streams from the recently created Cloud Watch Log groups (Cloud Trail Logs) were got copied over to the S3 bucket.

5. How to reduce the SubscribeSchedule: from Default: rate(1 hour) to 5 mins.
   Immensely appreciate your inputs to fix my issue.

[kichik]

Sounds like the Subscriber custom resource failed. The log for that should be in CloudWatch under something like /aws/lambda/CloudWatch2S3-LogSubscriberFunction-xxxx. Look for the group with LogSubscriberFunction in the name and please attach the logs from there.

If you have on account you can leave the accounts list as just 0.

Why do you want to reduce the subscription schedule to 5 minutes? Do you expect to be creating new log groups a lot? If you do, that's just a parameter for the stack that you can modify. Use rate(5 minute) instead of the default rate(1 hour).

[anithac76]

CloudWatch Logs from /aws/lambda/CloudWatch2S3-LogSubscriberFunction-xxxx:

04:39:26
no last exception: ValueError Traceback (most recent call last): File "/var/task/index.py", line 75, in handler traceback.print_last() File "/var/lang/lib/python3.6/traceback.py", line 173, in print_last raise ValueError("no last exception") ValueError: no last exception

no last exception: ValueError
Traceback (most recent call last):
File "/var/task/index.py", line 75, in handler
traceback.print_last()
File "/var/lang/lib/python3.6/traceback.py", line 173, in print_last
raise ValueError("no last exception")
ValueError: no last exception

[kichik]

Can you download the latest version and try again? It's not going to fix it, but it's going to hopefully print the real error. That one is just an error printing the error.

[anithac76]

Latest version of what should I download ?

[kichik]

Of this project's template.
https://github.com/CloudSnorkel/CloudWatch2S3/raw/master/CloudWatch2S3.template

[anithac76]

START RequestId: 885cad79-9446-479f-a1fd-c508f4c29c28 Version: $LATEST
event: {'RequestType': 'Create', 'ServiceToken': 'arn:aws:lambda:us-east-1:017396793107:function:CloudWatch2S3-LogSubscriberFunction-1NHU9F3DUQ2JV', 'ResponseURL': 'https://cloudformation-custom-resource-response-useast1.s3.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-east-1%3A017396793107%3Astack/CloudWatch2S3/7269a990-89d8-11ea-9821-0eba1c1b0c34%7CSubscriber%7C6f2ea364-446d-4442-9188-96a325fba81a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20200429T051642Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=AKIA6L7Q4OWTTDPGOFMG%2F20200429%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=914f493bd4858a79190524a4f30d4fe11e827ae1106a46d424bb1dc95ce89f78', 'StackId': 'arn:aws:cloudformation:us-east-1:017396793107:stack/CloudWatch2S3/7269a990-89d8-11ea-9821-0eba1c1b0c34', 'RequestId': '6f2ea364-446d-4442-9188-96a325fba81a', 'LogicalResourceId': 'Subscriber', 'ResourceType': 'Custom::Subscriber', 'ResourceProperties': {'ServiceToken': 'arn:aws:lambda:us-east-1:017396793107:function:CloudWatch2S3-LogSubscriberFunction-1NHU9F3DUQ2JV'}}
Subscribe to all new log groups on resource Create
Finding all log groups with prefix ''
Subscribe /aws/kinesisfirehose/CloudWatch2S3-DeliveryStream
Skipping our log groups to avoid endless recursion
Subscribe /aws/lambda/AddSubscription-AddSubscriptionLambda-12ZZMC05OVEGN
### Caught exception but unable to print stack trace
An error occurred (LimitExceededException) when calling the PutSubscriptionFilter operation: Resource limit exceeded.
https://cloudformation-custom-resource-response-useast1.s3.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-east-1%3A017396793107%3Astack/CloudWatch2S3/7269a990-89d8-11ea-9821-0eba1c1b0c34%7CSubscriber%7C6f2ea364-446d-4442-9188-96a325fba81a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20200429T051642Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=AKIA6L7Q4OWTTDPGOFMG%2F20200429%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=914f493bd4858a79190524a4f30d4fe11e827ae1106a46d424bb1dc95ce89f78
Response body:
{
"Status": "FAILED",
"Reason": "See the details in CloudWatch Log Stream: 2020/04/29/[$LATEST]8292d285c81e41619b779d7015c4bdb3",
"PhysicalResourceId": "fail",
"StackId": "arn:aws:cloudformation:us-east-1:017396793107:stack/CloudWatch2S3/7269a990-89d8-11ea-9821-0eba1c1b0c34",
"RequestId": "6f2ea364-446d-4442-9188-96a325fba81a",
"LogicalResourceId": "Subscriber",
"NoEcho": false,
"Data": {}
}
/var/runtime/botocore/vendored/requests/api.py:64: DeprecationWarning: You are using the put() function from 'botocore.vendored.requests'. This dependency was removed from Botocore and will be removed from Lambda after 2021/01/30. https://aws.amazon.com/blogs/developer/removing-the-vendored-version-of-requests-from-botocore/. Install the requests package, 'import requests' directly, and use the requests.put() function instead.
DeprecationWarning
Status code: OK
END RequestId: 885cad79-9446-479f-a1fd-c508f4c29c28
REPORT RequestId: 885cad79-9446-479f-a1fd-c508f4c29c28 Duration: 675.12 ms Billed Duration: 700 ms Memory Size: 128 MB Max Memory Used: 65 MB Init Duration: 262.17 ms

[anithac76]

[kichik]

Do you already have a subscription on /aws/lambda/AddSubscription-AddSubscriptionLambda-12ZZMC05OVEGN? I think that might be why it's failing.

It sounds like you don't want to subscribe to that log anyway as it's not coming from your web server. Maybe try setting "Required Log Group Name Prefix" to demoWebServer? This way only that log group will be exported, which sounds like what you want anyway.

[anithac76]

The only Log Group I have in CloudWatch logs is demoWebServer. Rest were all created by CloudWatch2S3. I deleted all of them except mine (demoWebServer) and tried again. I get the same error. I can't add a prefix because I want newer Log groups to be subscribed automatically when they are added and i do not know what their naming convention would be. demoWebServer is just my test log group which i configured to push my Webserver logs.

Please advise.

[kichik]

Are you sure? AddSubscriptionLambda is not a function that our template adds. It seems to be coming from a CloudFormation stack named AddSubscription. Can you also please include the latest Subscriber log from after you deleted the extra log groups?

You can also try the latest version. I made it skip log groups that are already subscribed to something else.

[kichik]

AddSubscriptionLambda comes from https://github.com/aws-samples/amazon-cloudwatch-log-centralizer/
You can't have both this project and that project installed at the same time. You should pick one that works better for you as they conflict.

[anithac76]

Voila, Create Stack Completed Successfully!
Couple of questions:

1. My demoWebserver Log group was already subscribed, so understandably it was not moved to S3. I have removed the Subscription filter to see if it will make it to S3 after 1 hour.

2. Aside, even the other three log groups created by CloudWatch2S3 did not make it to S3 Bucket. I see the S3 bucket cloudwatch2s3-logbucket-lb3dxaj5uwal created.
   But it is empty. This is the only thing Pending, which is the actual end goal!

[kichik]

demoWebserver will be subscribed in the next hour. Our logs don't get subscribed otherwise you'd get log recursion.

[anithac76]

Hi,
All issues with CF Stack Creation / setup is resolved now.

This morning I checked my S3 bucket and I see some logs organized in the folders such as below:
CloudWatch2S3-DeliveryStream-1A8AINC5ESAIQ-1-2020-04-29-08-01-36-4cb1740e-896f-4826-90f6-d5242713cfa4

I however do not see my Webserver logs captured under demoWebServer folder in CloudWatch anywhere in S3. Where will it be located ? Will the S3 bucket contain name of the CW Log group while capturing the raw data.
I will try creating a new log group and push some logs into it.

Also when I click on the Object URL of any entry in any of the other folders in S3, I get the below content for all of my entries in the log bucket. Where is it from ?

AccessDenied Access Denied 0745C4FE80374527 Ol+ygcTOnew4mQoJzypUuMoTSXhAewtViVdjSlqiT530VyRWGmJP5KUmhC95cRXrLoLWCnCXRgQ=

Thanks.

[kichik]

Don't click the URL. Click the Download link instead. This should let you see what's inside the logs in S3.

[anithac76]

Yes, I now see my raw data logs in S3. Thanks much for creating a great Tutorial and your help in issue resolution. Appreciate it!