-
Notifications
You must be signed in to change notification settings - Fork 582
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
iotune reports lower-than-expected IOPS on some large instances #17261
Comments
Perhaps it is not in fact totally I investigated a bit more on i3en.12xlarge, which had the following
(6xlarge also shown for reference: one would expect the 12xlarge numbers to all be double the 6x numbers) I focused only on read IOPS. Using I then tried creating an array of only the first two drives, this would presumably give identical results to 6x large, 500k R IOPS. However, it did not: in fact it only gave 1/2 the 4-drive output, i.e., the same performance probably was still evident here in the the same proportion as the 4 drive case. The next odd thing is that this effect seems to depend on which drives are paired up in the raid. If drives 1 and 2 are paired, or 3 and 4 are paired they are slow as described (~350k IOPS), but any other pairs result in a fast configuration (~475k IOPS, just slightly shy of the theoretical), shown here based on testing each combination:
This behavior was consistent and quite stable from run to run and confirmed on 2 different machines. Finally, I confirmed that this doesn't have anything to do with |
This isn't a special config: performance is similar under many parameter changes in the above: the main thing is that you need enough total queue depth (which is I didn't notice any changes in performance with increased file size, runtime or depth. |
Script to rebuild & remount arrays, using for testing all combination of array members: set -euo pipefail
echo "MD=${MD:=md0}"
echo "DEVICES=${DEVICES:=nvme1n1 nvme2n1}"
echo "MOUNT=${MOUNT:=/mnt/xfs}"
IFS=', ' read -r -a DA <<< "$DEVICES"
echo "DEVICE_COUNT=${#DA[@]}"
MDD=/dev/$MD
sudo umount /mnt/xfs* || true
sudo mdadm --stop /dev/md* || true
# set -x
sudo mdadm --create --run --verbose $MDD --level=0 --raid-devices=2 $(for d in $DEVICES; do echo -n "/dev/$d "; done)
sudo mkfs.xfs -f $MDD
sudo mkdir -p $MOUNT
sudo mount $MDD $MOUNT
sudo chmod a+w $MOUNT
echo "OK - mounted at: $MOUNT"
cat /proc/mdstat |
Miscellaneous node:
|
This issue hasn't seen activity in 3 months. If you want to keep it open, post a comment or remove the |
This issue was closed due to lack of activity. Feel free to reopen if it's still relevant. |
Version & Environment
Redpanda version: 23.3
What went wrong?
When running
rpk iotune
on large instance types, some results (especially IOPS) are often significantly lower than the vendor advertised numbers.See i3en for example:
Up to and include 6x large the values track closely the advertised IOPS. However 12xlarge reports 550k IOPS versus the advertised value of 1000k and 24xlarge ~1000k versus 2000k advertised.
I believe this is measurement/tuning error, not a fundamental hardware limitation.
This applies to other instance types as well, see #17220 for details.
What should have happened instead?
iotune
produces results reflecting the hardware capabilities.How to reproduce the issue?
rpk iotune
on i3en.12xlarge instances and observer outputJIRA Link: CORE-1915
The text was updated successfully, but these errors were encountered: