Today we ran some tests and benchmarks and while in the past with mechanical hard drives the hit from the overhead of running checksums etc for RAID5 and/or RAID6 was negligible, with SSD it became a factor and with the latest generation Optane disks the difference is just huge. Compare the following numbers taken on a 56 vCore server with 4x Intel Optane NVMe drives configured as RAID10, RAID5 and RAID0:
Server24 56 Cores with RAID10 NVMe, 256 GB RAM - 20201128
Sat Nov 28 11:04:33 CET 2020
Basic System Information:
---------------------------------
Processor : Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
CPU cores : 56 @ 1707.405 MHz
AES-NI : ✔ Enabled
VM-x/AMD-V : ✔ Enabled
RAM : 251.5 GiB
Swap : 4.0 GiB
Disk : 165.3 GiB
fio Disk Speed Tests (Mixed R/W 50/50):
---------------------------------
Block Size | 4k (IOPS) | 64k (IOPS)
------ | --- ---- | ---- ----
Read | 527.42 MB/s (131.8k) | 615.91 MB/s (9.6k)
Write | 528.81 MB/s (132.2k) | 619.15 MB/s (9.6k)
Total | 1.05 GB/s (264.0k) | 1.23 GB/s (19.2k)
| |
Block Size | 512k (IOPS) | 1m (IOPS)
------ | --- ---- | ---- ----
Read | 565.11 MB/s (1.1k) | 597.10 MB/s (583)
Write | 595.13 MB/s (1.1k) | 636.87 MB/s (621)
Total | 1.16 GB/s (2.2k) | 1.23 GB/s (1.2k)
iperf3 Network Speed Tests (IPv4):
---------------------------------
Provider | Location (Link) | Send Speed | Recv Speed
| | |
Clouvider | London, UK (10G) | 888 Mbits/sec | 650 Mbits/sec
Online.net | Paris, FR (10G) | 880 Mbits/sec | 533 Mbits/sec
WorldStream | The Netherlands (10G) | 913 Mbits/sec | 815 Mbits/sec
Biznet | Jakarta, Indonesia (1G) | 641 Mbits/sec | 40.5 Mbits/sec
Clouvider | NYC, NY, US (10G) | 833 Mbits/sec | 78.7 Mbits/sec
Velocity Online | Tallahassee, FL, US (10G) | 766 Mbits/sec | 138 Mbits/sec
Clouvider | Los Angeles, CA, US (10G) | 657 Mbits/sec | 69.6 Mbits/sec
Geekbench 5 Benchmark Test:
---------------------------------
Test | Value
|
Single Core | 934
Multi Core | 18812
Full Test | https://browser.geekbench.com/v5/cpu/5039076
https://github.com/masonr/yet-another-bench-script
curl -sL yabs.sh | bash
Server24 56 Cores with RAID5 NVMe, 256 GB RAM - 20201128
Sat Nov 28 11:50:37 CET 2020
Basic System Information:
---------------------------------
Processor : Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
CPU cores : 56 @ 1728.657 MHz
AES-NI : ✔ Enabled
VM-x/AMD-V : ✔ Enabled
RAM : 251.5 GiB
Swap : 4.0 GiB
Disk : 191.9 GiB
fio Disk Speed Tests (Mixed R/W 50/50):
---------------------------------
Block Size | 4k (IOPS) | 64k (IOPS)
------ | --- ---- | ---- ----
Read | 309.13 MB/s (77.2k) | 427.73 MB/s (6.6k)
Write | 309.95 MB/s (77.4k) | 429.99 MB/s (6.7k)
Total | 619.08 MB/s (154.7k) | 857.73 MB/s (13.4k)
| |
Block Size | 512k (IOPS) | 1m (IOPS)
------ | --- ---- | ---- ----
Read | 390.30 MB/s (762) | 506.25 MB/s (494)
Write | 411.04 MB/s (802) | 539.96 MB/s (527)
Total | 801.35 MB/s (1.5k) | 1.04 GB/s (1.0k)
iperf3 Network Speed Tests (IPv4):
---------------------------------
Provider | Location (Link) | Send Speed | Recv Speed
| | |
Clouvider | London, UK (10G) | busy | busy
Online.net | Paris, FR (10G) | 874 Mbits/sec | 505 Mbits/sec
WorldStream | The Netherlands (10G) | 906 Mbits/sec | 804 Mbits/sec
Biznet | Jakarta, Indonesia (1G) | 691 Mbits/sec | 42.5 Mbits/sec
Clouvider | NYC, NY, US (10G) | 721 Mbits/sec | 96.1 Mbits/sec
Velocity Online | Tallahassee, FL, US (10G) | 722 Mbits/sec | 180 Mbits/sec
Clouvider | Los Angeles, CA, US (10G) | 573 Mbits/sec | 63.1 Mbits/sec
Geekbench 5 Benchmark Test:
---------------------------------
Test | Value
|
Single Core | 929
Multi Core | 18811
Full Test | https://browser.geekbench.com/v5/cpu/5039758
Server24 56 Cores configured with RAID0 4x Intel Optane NVMe, 256 GB RAM - 20201128
Basic System Information:
---------------------------------
Processor : Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
CPU cores : 56 @ 2096.213 MHz
AES-NI : ✔ Enabled
VM-x/AMD-V : ✔ Enabled
RAM : 251.5 GiB
Swap : 4.0 GiB
Disk : 218.7 GiB
Preparing system for disk tests...curl -s -4 --connect-timeout 5 --retry 5 --retry-delay 0 https://raw.githubusercontent.com/masonr/yet-another-bench-script/master/bin/fio_x64 -o ./2020-11-28T12_06_42+01_00/disk/fio
fio Disk Speed Tests (Mixed R/W 50/50):
---------------------------------
Block Size | 4k (IOPS) | 64k (IOPS)
------ | --- ---- | ---- ----
Read | 768.58 MB/s (192.1k) | 1.11 GB/s (17.4k)
Write | 770.61 MB/s (192.6k) | 1.12 GB/s (17.5k)
Total | 1.53 GB/s (384.7k) | 2.23 GB/s (34.9k)
| |
Block Size | 512k (IOPS) | 1m (IOPS)
------ | --- ---- | ---- ----
Read | 1.08 GB/s (2.1k) | 1.07 GB/s (1.0k)
Write | 1.13 GB/s (2.2k) | 1.14 GB/s (1.1k)
Total | 2.22 GB/s (4.3k) | 2.21 GB/s (2.1k)
iperf3 Network Speed Tests (IPv4):
---------------------------------
Provider | Location (Link) | Send Speed | Recv Speed
| | |
Clouvider | London, UK (10G) | 875 Mbits/sec | 640 Mbits/sec
Online.net | Paris, FR (10G) | 886 Mbits/sec | 549 Mbits/sec
WorldStream | The Netherlands (10G) | 910 Mbits/sec | 799 Mbits/sec
Biznet | Jakarta, Indonesia (1G) | 651 Mbits/sec | 42.0 Mbits/sec
Clouvider | NYC, NY, US (10G) | 793 Mbits/sec | 103 Mbits/sec
Velocity Online | Tallahassee, FL, US (10G) | 728 Mbits/sec | 176 Mbits/sec
Clouvider | Los Angeles, CA, US (10G) | 450 Mbits/sec | 56.0 Mbits/sec
Geekbench 5 Benchmark Test:
---------------------------------
Test | Value
|
Single Core | 926
Multi Core | 18879
Full Test | https://browser.geekbench.com/v5/cpu/5040005
So we know for sure that RAID0 rules, but the sheer outperformance of RAID10 vs RAID5 clearly shows that with NVMe disks, the CPU becomes the limiting factor. And as my colleague Markus pointed out, probably this is the reason we see no RAID5 or RAID6 controllers on the market for NVMe disks – the sheer computing power of a CPU is barely enough to cope with SSDs.
Recent Comments