More r6 instance types available.

[BryanQuigley]

Detailed Description

We can upgrade from r5 (Intel) and maybe get better performance.

We previously evaluated r5a (AMD) and found it's performance lacking - and r5g was no where near the cost savings promised to be worth evaluating.

R6a marketing page highlights up to 35% improvement over r5a
R6i marketing page highlights up to 15% improvement over r5 (Intel)

					On Demand	Reserved
R6A Extra Large	r6a.xlarge	32.0 GiB	4 vCPUs	Up to 12.5 Gigabit	$165.5640 monthly	$109.5219 monthly
R5 Extra Large	r5.xlarge	32.0 GiB	4 vCPUs	Up to 10 Gigabit	$183.9600 monthly	$116.0700 monthly
R6I Extra Large	r6i.xlarge	32.0 GiB	4 vCPUs	Up to 12.5 Gigabit	$183.9600 monthly	$121.6910 monthly
R5A Extra Large	r5a.xlarge	32.0 GiB	4 vCPUs	Up to 10 Gigabit	$164.9800 monthly	$103.6600 monthly
R6G Extra Large	r6g.xlarge	32.0 GiB	4 vCPUs	Up to 10 Gigabit	$147.1680 monthly	$92.7100 monthly

Implementation

Note, staging will be a smaller instance type then listed here

1. Make fresh deployment on staging,
2. Get fresh benchmark numbers / https://github.com/PublicMapping/districtbuilder/tree/develop/load-tests and memory usage
3. Make terraform change to r6i and deploy to staging
4. Get r6i performance numbers
5. Make terraform change to r6a and deploy to staging
6. Get r6a performance numbers
7. Determine if we want to make an instance move. If no, document. If yes, continue:
8. Make terraform change in PR

[KlaasH]

Ok, I did a fair bit of testing and reached some conclusions.

Extremely short answer

We should switch to r6a.xlarge.

Medium-length answer

I think the limiting factor is single-threaded CPU performance, so for our purposes r5a < r5 < r6a < r6i. Since r6a is faster and cheaper than r5 (which is also Intel but the "i" is silent on that generation), it makes sense to switch to that. Switching to r6i would improve performance more, but by less than the difference between the r5 and the r6a, and it doesn't seem necessary if we're getting by right now.

Saying we're limited by CPU makes it seem like we should consider compute-optimized instances, but that's just because I didn't test different memory sizes. The app is routinely using 80% of the 32GB available to an r-series xlarge instance, and we'd need to go all the way to 4xlarge to get that much memory on a c-series instance. More importantly, since it's single-threaded performance that seems to be the issue (CPUUtilization never actually exceeds about 30% except during task initialization), throwing more vCPUs at it won't help, and the c6 instances use the same actual hardware as the r6 ones.

Relatedly, I didn't evaluate the r6g series because it crashed when we tried it before. I was going to say it doesn't seem likely that it would improve performance, since it's not built to be good at single-threaded speed, but it occurs to me that there's some chance that whatever single-threaded step is blocking us (per the same issue discussion, I think it's the topojson merge) could be implemented differently for ARM such that it's able to parallelize better. But I don't think it's a strong chance, and we'd have to build and publish ARM-based containers to find out.

Long answer

I used the load testing tooling added in PR #448 and updated in PR #1046 and PR #1172. Very conveniently, I was able to just use the exact same block of commands from the testing instructions for PR #1172. There are a lot of projects referenced in those commands, but I didn't have to recreate them because I was able to log into the database and steal all of them by updating all projects in the staging database that were owned by Mike to be owned by me instead. If we need to do this again, whoever is doing it should steal all my projects. Or we could consider making a shared testing user and setting the ownership to it.

Obviously API request performance is always noisy, and notably the different test cases also seem to vary a lot in difficulty--e.g. IL, NC, and MO were consistently pretty quick; PA, TX, and CA significantly slower--so my initial thought of looking at response time graphs and drawing conclusions didn't seem like a winner. They're interesting and not pure noise, but they're not different enough between instance types or consistent enough across runs of the same instance type to be super illuminating.

r5:

r6a:

r6i:

So instead of trying to get results visually, I saved all the statistics printed out by k6 and aggregated them. This is sloppy analysis, since it's taking averages of averages of averages, but I think it still produces results that are likely to represent the direction and a rough approximation of the magnitude of real effects.

Here's the bottom line from averaging the average and 95th-percentile response times reported by k6 across all runs for all states:

Instance	Combined avg	combined p95
r5	2.3561s	6.0505s
r6a	1.7658s	4.2495s
r6i	1.6486s	3.8121s

Here are some files related to that:

• The full k6 output logs, zipped: k6_output.zip
• The super-rough script that parses them and aggregates the results: parse_k6_results.txt (it's a Python script but Github won't let it be .py)
• The summary that the script prints out: db_app_summary.txt (apparently .json is verboten as well)
• The CSV file of the individual run results (i.e. each row has numbers from one http_req_duration line from the output of a k6 run for a particular state on a particular instance type): db_app_stats.csv

While I'm attaching files, here's the JSON for the dashboard I made: LoadTestingDashboard.txt (note: this isn't identical to the screenshots above because I accidentally deleted that one and had to recreate it)
Apparently the mere existence of a custom CloudWatch dashboard costs money, so I just deleted it. But it's hard to see what's going on when you have to flip between a bunch of different standard dashboards and metrics pages. This should make it easy to recreate it if we need to do more load testing in the future (just make a custom dashboard with nothing on it then go to "Actions>view/edit source" and copy the JSON in there).

Conclusion

Like the short answer said, we should switch to r6a. It's cheaper than r5 and possibly in the vicinity of 25% faster for our purposes. r6i is faster still, but it doesn't seem like the difference (roughly 10%) is enough to justify moving in the other direction price-wise.

[KlaasH]

I updated the production tfvars file with these changes:

53c53
< container_instance_type                          = "r5.xlarge"
---
> container_instance_type                          = "r6a.xlarge"
57c57
< container_instance_reserved_memory               = 1280
---
> container_instance_reserved_memory               = 1300

The second one is because one of the times I changed the instance type and triggered a deploy, no tasks actually came up because their requested memory allocation was slightly higher than the total memory available to be allocated to tasks. I'm not positive this was on the r6a, but I think it was.

The exact amount needed for overhead might change if we change OS version (per issue #1278), so it would make sense to revisit this setting once we've settled that question, to make sure we're allocating neither more nor less than all of the memory available on the instance.

[KlaasH]

Deployed with the above changes on 3/13/23.