So you have purchased a new VPS (whether it is with Binary Lane or another provider), logged in with SSH and are now staring at your root shell. For many of us, the first question that comes to mind is How fast is my server?; followed quickly by How do I measure its performance?


In this article I will look at some specific methods of measuring the disk performance of your VPS.


What not to do


Chances are, you have seen this test before; perhaps even used it yourself. It is the obligatory dd test - here is one popular variety:


dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync


This test is popular because dd is pre-installed on almost all Linux servers. While it is a simple way of seeing if something is broken (for example, if you see 10MB/sec than your server is overloaded) it has a number of problems:

  • This is a single-threaded, sequential-write test. If you are running the typical web+database server on your VPS, the number is meaningless because typical services do not do long-running sequential writes.
  • The amount of data written (1GB) is small; and hence can be strongly influenced by caching on the host server, or the host's RAID controller. (The conv=fdatasync only applies to the VPS, not the host).
  • It executes for a very short period of time; just a few seconds on faster I/O subsystems. This isn't enough to get a consistant result.
  • There's no read performance testing at all.

So if you accept my argument that dd is not a great way to benchmark disk performance, let's walk through a few tools that I think are worth using:

Measuring random IOPS with FIO


When running a website or similar workload, in general the best measurement of the disk subsystem is known as IOPS: Input/Output Operations per Second. In particular, the specific test we want to do is:
  1. Random reads, random writes, or a combination of both. Databases in particular will pull data from all over your disk - known as random access.
  2. 4 kilobyte blocks. Again, databases and many other programs will read very small chunks of data - 4 kilobytes is a good working estimate.
  3. Multiple threads. If your website has multiple visitors, your website will serve them all at the same time. We want our benchmark to simulate this behaviour of multiple things accessing the disk at once.
FIO is a popular tool for measuring IOPS on a Linux server. Its very configurable (perhaps even to its detriment) but with the following Bash snippets is easy enough to use.  To start with, here is how to download and compile it - just paste straight into the root shell of your CentOS/Debian/Ubuntu server:

  

cd /root
yum install -y make gcc libaio-devel || ( apt-get update && apt-get install -y make gcc libaio-dev  </dev/null )
wget https://github.com/Crowd9/Benchmark/raw/master/fio-2.0.9.tar.gz ; tar xf fio*
cd fio*
make

  

With FIO compiled, we can now run some tests.


Random read/write performance


If you simply want a way to compare different provider's disk performance, then this is the test I suggest. Run the following command:

   

./fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75

   

This will create a 4 GB file, and perform 4KB reads and writes using a 75%/25% (ie 3 reads are performed for every 1 write) split within the file, with 64 operations running at a time. The 3:1 ratio is a rough approximation of your typical database.


FIO will now tick along printing a summary as it goes, that looks like this:

  

Jobs: 1 (f=1): [m] [65.4% done] [161.5M/55417K /s] [41.4K/13.9K iops] [eta 00m:09s]

  

And eventually a full result output like this, with the numbers we want highlighted in yellow:


fio-2.0.9

Starting 1 process

Jobs: 1 (f=1): [m] [100.0% done] [164.4M/56926K /s] [42.8K/14.3K iops] [eta 00m:00s]

test: (groupid=0, jobs=1): err= 0: pid=32210: Wed Mar 13 16:16:04 2019

  read : io=3073.4MB, bw=191118KB/s, iops=47779 , runt= 16467msec

  write: io=1022.7MB, bw=63592KB/s, iops=15897 , runt= 16467msec

  cpu          : usr=21.29%, sys=69.69%, ctx=8450, majf=0, minf=4

  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%

     issued    : total=r=786784/w=261792/d=0, short=r=0/w=0/d=0


Run status group 0 (all jobs):

   READ: io=3073.4MB, aggrb=191117KB/s, minb=191117KB/s, maxb=191117KB/s, mint=16467msec, maxt=16467msec

  WRITE: io=1022.7MB, aggrb=63591KB/s, minb=63591KB/s, maxb=63591KB/s, mint=16467msec, maxt=16467msec


Disk stats (read/write):

  vda: ios=784951/261218, merge=0/23, ticks=241180/83160, in_queue=313312, util=92.35%


This tests shows that Binary Lane's SSD on this host performs 34596 read operations per second and 11527 write operations per second under a 75% read load, which is typical of SATA SSD. 



If you want to know what contributed to the above numbers, read on -

Random read performance


To measure random reads, use slightly altered FIO command:

 

./fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randread

  

Again, we pull the iops from the result:


fio-2.0.9

Starting 1 process

Jobs: 1 (f=1): [r] [100.0% done] [336.5M/0K /s] [86.2K/0  iops] [eta 00m:00s]

test: (groupid=0, jobs=1): err= 0: pid=32172: Wed Mar 13 16:11:11 2019

  read : io=4096.0MB, bw=342980KB/s, iops=85745 , runt= 12229msec

  cpu          : usr=21.43%, sys=76.05%, ctx=5387, majf=0, minf=68

  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%

     issued    : total=r=1048576/w=0/d=0, short=r=0/w=0/d=0


Run status group 0 (all jobs):

   READ: io=4096.0MB, aggrb=342980KB/s, minb=342980KB/s, maxb=342980KB/s, mint=12229msec, maxt=12229msec


Disk stats (read/write):

  vda: ios=1043901/20, merge=0/11, ticks=166560/4, in_queue=159628, util=96.92%



This tests shows Binary Lane's storage scoring 85745 read operations per second, a typical result for SATA SSD.  This is higher than our mixed read/write result because writes are not being performed, freeing up disk resources to execute the reads being tested.


Random write performance


Again, just modify the FIO command slightly so we perform randwrite instead of randread:


./fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randwrite

 

Again, we pull the iops from the result:


fio-2.0.9

Starting 1 process

Jobs: 1 (f=1): [w] [100.0% done] [0K/195.3M /s] [0 /49.1K iops] [eta 00m:00s]

test: (groupid=0, jobs=1): err= 0: pid=32203: Wed Mar 13 16:14:28 2019

  write: io=4096.0MB, bw=200790KB/s, iops=50197 , runt= 20889msec

  cpu          : usr=19.10%, sys=62.43%, ctx=8113, majf=0, minf=5

  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%

     issued    : total=r=0/w=1048576/d=0, short=r=0/w=0/d=0


Run status group 0 (all jobs):

  WRITE: io=4096.0MB, aggrb=200790KB/s, minb=200790KB/s, maxb=200790KB/s, mint=20889msec, maxt=20889msec


Disk stats (read/write):

  vda: ios=0/1040658, merge=0/25, ticks=0/487300, in_queue=452284, util=89.27%

 


This tests shows Binary Lane's SSD scoring 50197 write operations per second.


Measuring latency with IOPing


The final aspect of evaluating disk performance is to measure the latency on individual requests. One way to do so is to install ioping using the following command:

 

cd /root
yum install -y make gcc libaio-devel || ( apt-get update && apt-get install -y make gcc libaio-dev  </dev/null )
wget https://ioping.googlecode.com/files/ioping-0.6.tar.gz ; tar xf ioping*
cd ioping*
make

 

We can now run ioping to get an idea of the per-request latency:


 

./ioping -c 10 .

 

4096 bytes from . (ext3 /dev/vda1): request=1 time=0.2 ms

4096 bytes from . (ext3 /dev/vda1): request=2 time=0.3 ms 

4096 bytes from . (ext3 /dev/vda1): request=3 time=0.3 ms 

4096 bytes from . (ext3 /dev/vda1): request=4 time=0.3 ms 

4096 bytes from . (ext3 /dev/vda1): request=5 time=0.3 ms 

4096 bytes from . (ext3 /dev/vda1): request=6 time=0.3 ms 

4096 bytes from . (ext3 /dev/vda1): request=7 time=0.3 ms 

4096 bytes from . (ext3 /dev/vda1): request=8 time=0.2 ms 

4096 bytes from . (ext3 /dev/vda1): request=9 time=0.4 ms 

4096 bytes from . (ext3 /dev/vda1): request=10 time=0.3 ms 

 

--- . (ext3 /dev/vda1) ioping statistics --- 

10 requests completed in 9006.8 ms, 3505 iops, 13.7 mb/s 

min/avg/max/mdev = 0.2/0.3/0.4/0.1 ms 


Here you can see the average was 0.3ms.  On a healthy system, you should see relatively low variation and an average below 1.0 ms.


Final Thoughts


I hope this article has added a few more tools to your toolbelt and if nothing else, helped to dispell the idea that dd should be used for benchmarking.