Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rpk: add newer EC2 instance types #17220

Merged
merged 1 commit into from
Mar 25, 2024
Merged

rpk: add newer EC2 instance types #17220

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
136 changes: 136 additions & 0 deletions src/go/rpk/pkg/tuners/iotune/data.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ func ToJSON(props IoProperties) (string, error) {
}

func precompiledData() map[string]map[string]map[string]io {
// see IoProperties for field mapping
return map[string]map[string]map[string]io{
"aws": {
"i3.large": {
Expand Down Expand Up @@ -114,6 +115,141 @@ func precompiledData() map[string]map[string]map[string]io {
"i3en.metal": {
"default": {"", 257024 * 8, 2043674624 * 8, 174080 * 8, 1024458752 * 8},
},

// is4gen values are taken from measurements except where noted.
// For reference, the advertised IOPS values also also shown, these are from:
// https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/storage-optimized-instances.html
// Except for is4gen.8xlarge the alignment between advertised and measured IOPS is very close.
// Test results on is4gen show that the BW/IOPS ratio is 8K for reads and 5K for writes.
"is4gen.medium": {
// advertised 31250 25000
"default": {"", 31407, 251665056, 25105, 125274040},
},

"is4gen.large": {
// advertised 62500 50000
"default": {"", 62804, 503442048, 50214, 251087344},
},

"is4gen.xlarge": {
// advertised 125000 100000
"default": {"", 125659, 1015560064, 100404, 502173344},
},

"is4gen.2xlarge": {
// advertised 250000 200000
"default": {"", 251076, 2066053248, 201105, 1012893056},
},

"is4gen.4xlarge": {
// advertised 500000 400000
"default": {"", 502117, 4132124160, 402148, 2025763456},
},

// The measured values for is4gen.8xlarge broke the expected pattern
// and compared to 4xlarge for IOPS were slightly lower (!!) on the read side and
// only slightly higher on the write side, rather than the expected 2x
// jump. Signs point towards a measurement limitation rather than a real
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For my own understanding, what would be the theoretical impact if it is not a measurement limitation?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well for one it would mean that the instances cannot achieve their advertised performance, which is something to bring up with the cloud vendor I guess. From a practical impact it means that the performance ceiling would be lower than expected on those instance types, but for the most part the gap was small enough that it doesn't necessarily impact performance at the levels expected in our tiers.

Of course I wouldn't expect anyway to simply accept my hand-wavy explanation that this is measurement error and "probably fine" indefinitely :), so here's the follow up: #17261

// limitation, so for now we use 2x the prior instance type values.
// See: https://github.com/redpanda-data/redpanda/issues/17162.
"is4gen.8xlarge": {
// advertised 1000000 800000
// measured 457957, 7433367040, 433330, 4051469824
"default": {"", 2 * 502117, 2 * 4132124160, 2 * 402148, 2 * 2025763456},
},

"im4gn.large": {
// advertised 31250 25000
"default": {"", 31407, 251664976, 25105, 125273408},
},

"im4gn.xlarge": {
// advertised 62500 50000
"default": {"", 62809, 503439040, 50214, 251087600},
},

"im4gn.2xlarge": {
// advertised 125000 100000
"default": {"", 125664, 1015557248, 100411, 502192960},
},

"im4gn.4xlarge": {
// advertised 250000 200000
"default": {"", 251063, 2066157056, 201088, 1012886080},
},

"im4gn.8xlarge": {
// advertised 500000 400000
"default": {"", 502106, 4132167424, 402129, 2025818624},
},

// see comment on is4gen.8xlarge for what's going on here
"im4gn.16xlarge": {
// advertised 1000000 800000
// measured 556572, 8128993280, 497509, 4051727616},
"default": {"", 2 * 502106, 2 * 4132167424, 2 * 402129, 2 * 2025818624},
},

// i4i values are taken from direct measurement with iotune except
// where noted.
// i4i has a ~7K read BW/IOPS ratio and a 10K write BW/IOPS ratio,
// where the write side is notably higher than other storage optimized
// instance types.
"i4i.large": {
// advertised: 50000 27500
"default": {"", 50203, 352041984, 27599, 275442496},
},

"i4i.xlarge": {
// advertised: 100000 55000
"default": {"", 100371, 707373312, 55269, 552871552},
},

"i4i.2xlarge": {
// advertised: 200000 110000
"default": {"", 200633, 1427463808, 110542, 1113520256},
},

// at 4xlarge and above the read-side IOPS values start to scale
// more poorly that advertised/expected, so use scaled values
// from 2xlarge instead, under the assumption it is a measurement
// limitation
"i4i.4xlarge": {
// advertised: 400000 220000
// measured: 302635, 2853325312, 221201, 2259740160,
"default": {"", 2 * 200633, 2 * 1427463808, 2 * 110542, 2 * 1113520256},
},

"i4i.8xlarge": {
// advertised: 800000 440000
// measured 519944, 5704762368, 442395, 4519457792},
"default": {"", 4 * 200633, 4 * 1427463808, 4 * 110542, 4 * 1113520256},
},

// 12x large was not measured
"i4i.12xlarge": {
// advertised: 1200000 660000
"default": {"", 6 * 200633, 6 * 1427463808, 6 * 110542, 6 * 1113520256},
},

"i4i.16xlarge": {
// advertised: 16000000 880000
// measured 752697, 8585068032, 750060, 8211869696},
"default": {"", 8 * 200633, 8 * 1427463808, 8 * 110542, 8 * 1113520256},
},

// 24x large was not measured
"i4i.24xlarge": {
// advertised: 2400000 1320000
"default": {"", 12 * 200633, 12 * 1427463808, 12 * 110542, 12 * 1113520256},
},

// 32x large was not measured, iotune fails to start
// see https://github.com/redpanda-data/redpanda/issues/17154
"i4i.32xlarge": {
// advertised: 3200000 1760000
"default": {"", 16 * 200633, 16 * 1427463808, 16 * 110542, 16 * 1113520256},
},
},
}
}
Loading