Failed to deploy, function code combined with layers exceeds the maximum allowed size of 262144000 bytes

[LazyBrush]

Describe the bug
Using nestjs with sls to deploy a lambda with otel. I'm using the collector layer to then shuffle telemetry to a S3 bucket for later offline inspection.

When I deploy without the opentelemetry-collector-amd64-0_13_0:1 image layer the AWS console for the lambda says the bundle size is 54.7MB, when I add the layer sls fails to deploy with the error maximum allowed size of 262144000 bytes, the actual size is 264975312 bytes.

If we believe these numbers then it looks like the collector layer is about 210MB or over 80% of max lambda size, some of the documentation says 100MB+

Steps to reproduce
nestjs app as lambda, using otel, deployed with serverless (sls).

What did you expect to see?
My app deployed, nice otel tracing and my heart beating with joy.

What did you see instead?
sls failed to deploy, me crying inside.

What version of collector/language SDK version did you use?
xxx:layer:opentelemetry-collector-amd64-0_13_0:1

What language layer did you use?
Perhaps I am not familiar with the language layer, my guess this give auto instrumentation that wraps your actual code. Instead, I've build my nestjs app with otel sdk, so I am creating spans/metrics/logs manually, so my guess is I do not need a language layer?

Additional context
I can run the nestjs code locally with a docker compose collector and the suite of tools like jaeger, prometheus, etc.

I tried to use webpack to reduce the image size.

I also went down the path of trying to use a container for the lambda, but then layers are not supported, so I would not have the collector layer to shuffle the telemetry to S3.

My main aim is to have a minimal cost solution but still have observability. My thoughts were I could run a lambda without the traditional jaeger/prom/etc tools, just write to S3 logs and then offline or when there was an issue I could spin up those tools on my dev machine to then inspect data in S3.

Happy to have suggestions how best to proceed

Thank you!

[wpessers]

The collector layer package size being that big does not seem realistic to me. I just checked the size for the latest version of the collector layer after building it locally.

warre@PCWARRE:~/development$ unzip opentelemetry-collector-layer-amd64.zip 
Archive:  opentelemetry-collector-layer-amd64.zip
   creating: collector-config/
  inflating: collector-config/config.yaml  
   creating: extensions/
  inflating: extensions/collector    
warre@PCWARRE:~/development$ du -b collector-config
4471    collector-config
warre@PCWARRE:~/development$ du -b extensions/
44970136        extensions/

This is output in bytes, so total would be 44974607 bytes or ~43KiB

Are there any other details about your setup we should know about? Are the otel layers the only ones you use? Or do you have other additional layers?

[LazyBrush]

Hi wpessers,

Thanks for replying,

Very interesting, and that 44Meg is unzipped, I think AWS have a 250Meg limit of the zip size.

As you say, something doesn't add up. More/Recap details...

Otel collector is the only layer I have.
It's a Nestjs monorepo app (typescript), using serverless to deploy.
Code uses the AWS V3 SDK for DynamoDB,S3.
Serverless also adds roles (for S3 access,etc).
The nestjs app is just 1 lambda function.
Serverless links this to my domain and certificate, so I get a https endpoint with nice url.
After 'nest build' my ./dist/ folder is about 11Meg

aws lambda get-layer-version-by-arn --arn  arn:aws:lambda:eu-west-2:184161586896:layer:opentelemetry-collector-amd64-0_12_0:1

Gives "CodeSize": 13907741

Likewise, when I get info on my deployed lambda WITHOUT the layer, I see "CodeSize": 57307340, zipped

I just tried manually adding the layer via the aws lambda console to this function, same error.

--
This page https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html
Says 250MB for UNZIPPED package. Dang.

I guess this is very likely the node_modules pushing over the limit.

It's a shame that neither serverless nor aws help you here 'I see your node modules are too big, aws node base image comes with x y and z and you don't need to have them in your bundle'.

It's like almost free lambda is a gateway drug to spending more and more money on aws. Any more than a hello world and things get tricksie.

There is a certain irony to the fact that trying to add Observability to my code falls over due to the hidden and obscure ways that aws works.

[wpessers]

So are you bundling your code using something like esbuild or webpack? How big is your own deployment package after bundling (before compression)?

[serkan-ozal]

Hi @LazyBrush,

I think, even though your Lambda function zipped size (without OTEL collector layer) is 54.7 MB, it is very close to 250 MB after unzipped. 250 MB Lambda artifact size limit is the maximum function code size limit including with layer after unzip.

To verify that, can you measure the unzipped size of your Lambda function artifact?

[serkan-ozal]

Hi @LazyBrush,

Had you a chance to check my suggestion?

[LazyBrush]

Hello all,

Sorry for the delay, many plates to spin.

Yes, webpack. Nestjs uses webpack to bundle, so the dist/app js is about 11Meg. The node_modules is about 400Meg, but that is also devDependencies. Not sure how I see the exact size of the deployment package after bundling?

Yes, I would say this is exactly the problem that the deployment is too big for a lambda.

I don't think I'm doing too much in the code, nestjs, doing some AWS dynamodb calls, doing some JWT verification, and a handful of endpoints for different work with some telemetry.

Given all this, is there other approaches I can take? Main requirements are cost and observability.

We wanted to go with lambda as until there is good number of users, we don't want the cost of EC2s etc.
We liked the method of just storing all the telemetry into S3 via the collector, so it's all static until we want to look at it, or use the S3 for batch processing. So no need for other running telemetry tools 24/7.

I did try to use an ECS type lambda, which does not have this 250Meg limit, but they do not support the layers, so can't do use this 'opentelemetry-lambda' style layers method.

Thanks again for the replies

[serkan-ozal]

Hi @LazyBrush,

Does your final bundle (dist/app js) include dev dependencies too? If so, can you be sure that dev dependencies are ignored during bundling.

[LazyBrush]

Hi all,

An update, I've got a setup with works now. This is good for me, but it's not using the opentelemetry-lambda, sorry!

I still have the lambda with it's own nest-otel library which does the telemetry outward calls. But rather than those calls going to the opentelemetry-lamdba layer and then onward, I'm using an off the shelf service by Dash0 which I can hit and comes with quite a nice UI and costs are comparible with lamdba style pricing. It is always on and gives me a better price/functionality solution than storing to S3 and post processing the telemetry.

Thanks again, we can close this issue.