Investigate effectiveness of keep_warm vs standard AWS Lambda cold start solutions

[monkut]

Summary

Zappa's keep_warm feature uses a CloudWatch Events rule (default: rate(4 minutes)) to periodically invoke the Lambda function via handler.keep_warm_callback, keeping one execution environment warm. While this has been a core feature since early versions, AWS has since introduced native cold start mitigation options that may be more effective, cost-efficient, or complementary.

This issue proposes investigating the real-world effectiveness of keep_warm and comparing it against current AWS-native solutions.

Current Implementation

• keep_warm (default: true): Schedules a CloudWatch event to invoke the Lambda on a timer
• keep_warm_expression (default: rate(4 minutes)): Controls invocation frequency
• keep_warm_callback: Calls lambda_handler with an empty event to trigger web app initialization
• Only keeps one execution environment warm — concurrent requests still hit cold starts

AWS-Native Cold Start Solutions to Compare

1. Provisioned Concurrency (GA since Dec 2019)

• Pre-initializes a specified number of execution environments
• Eliminates cold starts for up to N concurrent requests
• Costs: ~$0.015/GB-hour (provisioned) + reduced per-request cost
• Can be combined with Application Auto Scaling

2. SnapStart (GA for Java since Nov 2022, Python since Nov 2024)

• Takes a snapshot of the initialized execution environment
• Restores from snapshot instead of full cold boot
• Reduces cold start from seconds to ~200-400ms for Python
• No additional cost beyond standard Lambda pricing

3. Lambda Function URLs / Response Streaming

• Not directly a cold start solution, but architectural alternatives that may affect warm behavior

4. ARM64 / Graviton2

• Faster init time and lower cost — can indirectly reduce cold start impact

Investigation Areas

• Measure actual cold start times with and without keep_warm for typical Zappa apps (Flask, Django, FastAPI) — Cold init ~1,020-1,080ms across all frameworks (512MB, Python 3.12). No difference between keep_warm on/off for cold start duration.
• Quantify the gap keep_warm leaves: concurrent cold starts, environment recycling despite warm pings — keep_warm only keeps 1 env warm. Cold start penalty is identical (~1,050ms init) when it occurs. keep_warm reduces frequency, not severity.
• Cost comparison: CloudWatch event invocations vs Provisioned Concurrency vs SnapStart — keep_warm ~$0.05/mo, SnapStart $0.00, Provisioned Concurrency ~$5.53/mo per env. See detailed comment.
• SnapStart compatibility: Can Zappa leverage Python SnapStart? What changes would be needed? — Yes, Zappa already has snap_start setting but it has two bugs: snap_start setting not passed to create_lambda_function on initial deploy #1447 (not passed on initial deploy) and SnapStart requires version publish after config update, but zappa update publishes before enabling #1448 (version published before config applied). SnapStart reduces cold start by 45-72% (Flask 371ms, Django 281ms, FastAPI 590ms restore vs ~1,050ms init).
• Provisioned Concurrency integration: Should Zappa offer a provisioned_concurrency setting as an alternative or complement to keep_warm? — Yes. Provisioned Concurrency eliminates cold starts entirely (0ms init, 2-5ms warm duration). Zappa should add a setting that creates aliases and configures provisioned concurrency.
• Hybrid approach: Would keep_warm + SnapStart or keep_warm + Provisioned Concurrency yield better results than any single approach? — keep_warm + SnapStart is the best cost/benefit hybrid (~$0.05/mo): fewer cold starts + faster when they occur. Provisioned Concurrency is best for latency-sensitive production apps.
• Documentation: Update guidance on when to use which approach based on traffic patterns — Recommendations: hobby/low-traffic → keep_warm + SnapStart; production → Provisioned Concurrency with auto-scaling; cost-sensitive production → keep_warm + SnapStart.

Expected Outcome

A clear recommendation (with data) on whether:

1. keep_warm remains the best default for Zappa users
2. Zappa should add first-class support for Provisioned Concurrency and/or SnapStart
3. keep_warm should be deprecated in favor of native AWS solutions
4. A combination approach provides the best experience

Result: See investigation comment. All four questions answered: (1) yes, keep_warm remains a good default; (2) yes, fix SnapStart bugs and add Provisioned Concurrency support; (3) no, don't deprecate keep_warm — it's complementary; (4) keep_warm + SnapStart is the recommended default combination.

Related

• keep_warm setting in zappa_settings.json
• handler.keep_warm_callback in zappa/handler.py
• CloudWatch Events scheduling in zappa/cli.py
• snap_start setting not passed to create_lambda_function on initial deploy #1447 — snap_start not passed on initial deploy
• SnapStart requires version publish after config update, but zappa update publishes before enabling #1448 — SnapStart version ordering bug during update

[monkut]

Cold Start Investigation Results

Test Setup

• Region: us-east-1
• Runtime: Python 3.12 (x86_64)
• Memory: 512 MB
• Frameworks: Flask (minimal), Django (minimal), FastAPI (minimal ASGI)
• Zappa version: 0.62.1 (current master)
• Trials: 5 per configuration, forced cold starts via env var update
• Configurations tested:
  • baseline — keep_warm: true (default Zappa behavior)
  • nowarm — keep_warm: false (no warm ping)
  • snapstart — snap_start: "PublishedVersions" (SnapStart enabled, published version)
  • provisioned — Provisioned Concurrency = 1 (via alias)

Cold Start Measurements

Framework	Config	Cold Start %	Avg Init (ms)	Avg Duration (ms)	Memory (MB)
Flask	baseline (keep_warm)	100%*	1,078	1,227	91
Flask	nowarm	100%*	1,070	1,222	92
Flask	snapstart	20%**	371 (restore)	265	85
Flask	provisioned	0%	0	5.4	92
Django	baseline (keep_warm)	100%*	1,022	2,816	103
Django	nowarm	100%*	1,018	2,907	103
Django	snapstart	20%**	281 (restore)	598	102
Django	provisioned	0%	0	586	103
FastAPI	baseline (keep_warm)	100%*	1,074	3,137	105
FastAPI	nowarm	100%*	1,060	3,090	105
FastAPI	snapstart	20%**	590 (restore)	678	97
FastAPI	provisioned	0%	0	5.3	105

* 100% cold because tests force cold starts via env var updates. In production, keep_warm prevents most cold starts for serial requests.
** 20% = 1 of 5 trials hit SnapStart restore; rest were warm hits on the same environment. SnapStart restore times shown are for the cold starts only.

Key Findings

1. keep_warm vs no warm: identical cold start behavior

Baseline (keep_warm=true) and nowarm show identical cold start init times (~1,020-1,080ms across all frameworks). The keep_warm feature doesn't reduce cold start duration — it reduces cold start frequency by keeping one execution environment alive. When a cold start does happen (concurrent requests, environment recycling), the penalty is the same.

2. SnapStart significantly reduces cold start time

SnapStart restore durations vs normal cold init:

• Flask: 371ms vs 1,078ms (66% reduction)
• Django: 281ms vs 1,022ms (72% reduction)
• FastAPI: 590ms vs 1,074ms (45% reduction)

SnapStart is most effective for Django (largest reduction) likely because Django's import chain benefits most from snapshot restore.

3. Provisioned Concurrency eliminates cold starts entirely

With provisioned concurrency, zero cold starts across all trials. Warm response times were 2-17ms for Flask/FastAPI and 2-5ms for Django (after initial request).

4. Memory usage

SnapStart uses slightly less memory (85-102MB vs 91-105MB), likely because the snapshot restore skips some initialization overhead.

Cost Comparison (monthly, per function, us-east-1)

Approach	Monthly Cost	Cold Start Eliminated?	Notes
keep_warm (default)	~$0.05	Partially (1 env only)	10,800 CloudWatch events + Lambda invocations
SnapStart	$0.00	No, but 45-72% faster	No additional cost; part of standard Lambda pricing
Provisioned Concurrency (1 env)	~$5.53	Yes (for 1 concurrent)	$0.015/GB-hour × 0.5GB × 720h = $5.40 + events
Provisioned Concurrency (5 env)	~$27.65	Yes (for 5 concurrent)	Scales linearly; can use auto-scaling
keep_warm + SnapStart	~$0.05	Best of both: fewer cold starts + faster when they happen	Recommended hybrid

Bugs Discovered

During this investigation, two bugs were found in Zappa's SnapStart implementation:

1. snap_start setting not passed to create_lambda_function on initial deploy #1447 — snap_start setting not passed to create_lambda_function on initial deploy. The setting is only applied during zappa update, not zappa deploy.

2. SnapStart requires version publish after config update, but zappa update publishes before enabling #1448 — During zappa update, the version is published before SnapStart config is applied, so the published version never gets a SnapStart snapshot. Users must run zappa update twice or manually publish a version.

Recommendations

1. keep_warm remains a good default — at ~$0.05/month, it's the cheapest way to reduce cold start frequency for low-traffic apps with serial request patterns.

2. Add first-class SnapStart support — SnapStart is free and reduces cold start duration by 45-72%. Zappa should:

   • Fix snap_start setting not passed to create_lambda_function on initial deploy #1447 and SnapStart requires version publish after config update, but zappa update publishes before enabling #1448
   • Automatically publish a version after enabling SnapStart
   • Document that SnapStart + keep_warm is the recommended hybrid approach
   • Consider making snap_start: "PublishedVersions" the default (it has no cost and no downside for Python 3.12+)

3. Add provisioned_concurrency setting — For latency-sensitive apps, Zappa should support configuring provisioned concurrency directly in zappa_settings.json, including:

   • Creating/managing aliases pointing to published versions
   • Setting provisioned concurrency on the alias
   • Optional auto-scaling configuration

4. Don't deprecate keep_warm — It serves a different purpose (maintaining warm environments) vs SnapStart (faster cold starts) vs provisioned concurrency (guaranteed warm environments). They're complementary.

5. Recommended configurations by use case:

   • Low traffic / hobby projects: keep_warm: true + snap_start: "PublishedVersions" (~$0.05/month)
   • Production APIs: Provisioned Concurrency with auto-scaling ($5-28+/month)
   • Cost-sensitive production: keep_warm: true + snap_start: "PublishedVersions" (best bang for buck)

Test Artifacts

All test projects used minimal framework apps (Flask/Django/FastAPI) with a single / endpoint returning JSON. Tests forced cold starts by updating Lambda environment variables. SnapStart was tested against published versions with confirmed OptimizationStatus: On.

[monkut]

Cost Analysis Correction: SnapStart Has Charges

The original investigation stated SnapStart was free. After querying the AWS Pricing API directly, SnapStart has two separate charges that significantly affect the cost comparison.

Verified AWS Pricing (us-east-1, x86, from Pricing API 2026-04-02)

Component	Rate	Unit
Standard Lambda duration	$0.0000166667	per GB-second
Standard Lambda requests	$0.0000002	per request
Provisioned Concurrency allocation	$0.0000041667	per GB-second (always-on)
Provisioned Concurrency invocation	$0.0000097222	per GB-second (replaces standard)
SnapStart cache	$0.0000015046	per GB-second (continuous while version exists)
SnapStart restore	$0.0001397998	per GB (per cold start restore)

Free tier: 1M requests + 400K GB-seconds/month. Provisioned Concurrency and SnapStart charges have no free tier.

Actual Test Cost (this investigation)

Component	Cost	Details
Lambda requests & duration	$0.00	~162 invocations, ~49 GB-sec (within free tier)
Provisioned Concurrency	$0.02	3 functions × 0.5GB × 1hr allocation
SnapStart cache	$0.01	3 functions × 0.5GB × 1.5hr cached
SnapStart restores	$0.00	3 restores × 0.5GB
Total	$0.03

Corrected Monthly Cost Projections (per 512MB function)

Low traffic (1,000 requests/day)

Approach	Monthly Cost	Cold Starts/mo	Notes
keep_warm only	$0.00	1,500 (~5%)	Everything in free tier
keep_warm + SnapStart	$2.05	1,500 (faster)	Cache $1.95 + restores $0.10
SnapStart only	$2.37	6,000 (~20%)	Cache $1.95 + restores $0.42
Provisioned Concurrency	$5.40	0	$5.40 allocation

Medium traffic (10,000 requests/day)

Approach	Monthly Cost	Cold Starts/mo	Notes
keep_warm only	$0.00	6,000 (~2%)	Still within free tier
keep_warm + SnapStart	$2.37	6,000 (faster)	Cache $1.95 + restores $0.42
SnapStart only	$4.05	30,000 (~10%)	More restores = higher cost
Provisioned Concurrency	$5.41	0	$5.40 + small invocation cost

High traffic (100,000 requests/day)

Approach	Monthly Cost	Cold Starts/mo	Notes
keep_warm only	$0.40	30,000 (~1%)	Exceeds free tier slightly
keep_warm + SnapStart	$4.45	30,000 (faster)	Cache $1.95 + restores $2.10
Provisioned Concurrency	$5.87	0	Best value at high traffic
SnapStart only	$12.83	150,000	Restore charges add up fast

Key Takeaway: SnapStart Cache is ~$1.95/month (512MB function)

The SnapStart cache charge of $0.0000015046/GB-second is continuous for as long as a SnapStart-enabled published version exists. For a 512MB function:

0.5 GB × 2,592,000 sec/month × $0.0000015046 = $1.95/month

This is ~36% of what Provisioned Concurrency costs ($5.40/month for 512MB × 1 env), but far from free. The restore charge ($0.0001397998/GB per restore) also scales with cold start frequency.

Revised Recommendations

1. Hobby / low-traffic: keep_warm: true alone remains the cheapest option at $0.00/month (free tier). SnapStart adds $2+/month for faster cold starts — worth it only if sub-second cold starts matter.

2. Medium traffic, cost-sensitive: keep_warm: true alone is still $0.00/month. Add SnapStart only if the user experience impact of ~1s cold starts justifies ~$2.40/month.

3. High traffic or latency-critical: Provisioned Concurrency ($5.40+/month) becomes the best value because it eliminates cold starts entirely and its cost doesn't scale with cold start frequency (unlike SnapStart restores).

4. SnapStart without keep_warm is the worst option at all traffic levels — more frequent cold starts mean more restore charges, and the cache charge is still $1.95/month regardless.