CentOS 6 Kvm Disk Write Performance

Home » CentOS » CentOS 6 Kvm Disk Write Performance
CentOS, CentOS-Virt 7 Comments

I have 2 similar servers. Since upgrading one from CentOS 5.5 to 6, disk write performance in kvm guest VMs is much worse.

There are many, many posts about optimising kvm, many mentioning disk performance in CentOS 5 vs 6. I’ve tried various changes to speed up write performance, but northing’s made a significant difference so far:

  • Install virtio disk drivers in guest
  • update the host software
  • Update RAID firmware to latest version
  • Switch the host disk scheduler to deadline
  • Increase host RAM from 8GB to 24GB
  • Increase guest RAM from 2GB to 4GB
  • Try different kvm cache options
  • Switch host from ext4 back to ext3
  • Set noatime on the virtual disk image file Note: There is no encryption or on-access virus scanner on any host or guest.

Below are some the block write figures in MB/s from bonnie++ with various configurations:

First, figures for the hosts show that the CentOS 6 server is faster:

54 CentOS 5 Host
50 CentOS 5 Host
69 CentOS 6 host
70 CentOS 6 host

Figures for a CentOS 6 guest running on the CentOS 5 host show that the performance hit is less than 50%:

30 CentOS 6 guest on CentOS 5 host with no optimisations
27 CentOS 6 guest on CentOS 5 host with no optimisations
32 CentOS 6 guest on CentOS 5 host with no optimisations

Here are the figures a CentOS 6 guest running on the CentOS 6 host with various optimisations. Even with these optimisations, performance doesn’t come close to the un-optimised guest running on the CentoOS 5 host:

5 No optimisations (i.e. same configuration as on CentOS 5)
4 deadline scheduler
5 deadline scheduler
15 noatime,nodiratime
14 noatime,nodiratime
15 noatime
15 noatime + deadline scheduler
13 virtio
13 virtio
10 virtio + noatime
9 virtio + noatime

The CentOS 6 server has a better RAID card, different disks and more RAM, which might account for the better CentOS 6 host performance. But why might the guest write performance be so much worse?

Is this a known problem? If so, what’s the cause? If not, is there a way to locate the problem rather than using trial and error?

Thanks, Julian

7 thoughts on - CentOS 6 Kvm Disk Write Performance

  • How are you running KVM machines? Through libvirt? Have you made any changes to the cgroups? What are you using as disks, raw files, lvm?

  • Nux! wrote:

    Not through libvirt, but using the qemu-kvm command. This is the full command-line (without libvirt):

    /usr/libexec/qemu-kvm -k en-gb -m 4096 -smp 2 -pidfile
    /media/archive/vm022/.vm022.pid -monitor unix:/media/archive/vm022/.vm022.monitor,server,nowait -usb -usbdevice tablet -net nic,macaddr:00:01:00:24:02 -net tap,script=/etc/incharge/qemu-ifup -vnc :2 -drive file=/media/vm022/hda.raw,index=0,media=disk -drive file=/media/archive/vm022/hdb.raw,index=1,media=disk -drive file=/media/archive/vm022/hdc.raw,index=2,media=disk

    > Have you made any changes to the cgroups?

    No, I never heard of cgroups until today!

    > What are you using as disks, raw files, lvm

    There are 2 physical disks in raid 1 configuration Apart from system and swap partitions, the majority of the disk is LVM
    Each VM has 3 virtual disk files. 1 system disk (hda), one temp disk
    (hdb) and one spare (hdc). All are raw format. The hda for each VM is stored in its own LVM volume, so a snapshot can be made and mirrored with minimal overhead. So the volume /media/vm022 contains one file (hda.raw) and one folder which is used by a monitoring task to check the volume hasn’t gone read-only. hdb, hdc and the monitor & pid files are stored on a larger LVM volume
    /media/archive There was no LVM snapshot while any of the test timings were made.

    As there are so few files on /media/vm022, the reserved space is reduced using tune2fs -r 73134 /dev/mapper/vm-ic022
    …so the 32GB file fits into a 33GB LVM volume.
    (See https://www.CentOS.org/modules/newbb/viewtopic.php?topic_id0854)

    All the above is identical on both the CentOS 5 and CentOS 6 servers, except that ext3 is used on the CentOS 5 host and ext4 on the CentOS 6
    host, and the disk & raid hardware is a different brand. In all cases, the guest VM is CentOS 6.2 and uses ext4.

  • >> I have 2 similar servers. Since upgrading one from CentOS 5.5 to 6, disk write performance in kvm guest VMs is much worse.

    Philip Durbin wrote:
    > Nice post, Julian. It generated some feedback at http://irclog.perlgeek.de/crimsonfu/2012-08-10 and a link to http://rhsummit.files.wordpress.com/2012/03/wagner_network_perf.pdf
    > Phil

    Thanks Phil for linking to my post on #crimsonfu, and reporting the result back here.

    In response to the points jimi_c raised there:

    > I don’t see his test numbers for different caching options

    All the test figures I gave are using cache=writethrough. cache=writeback produced much better figures than even the host (about
    180MB/s), because it is really writing to memory, but I don’t think it’s a safe option to use. cache=none produced worse figures. I didn’t include the figures because I did those tests before I started using bonnie++ (I was just timing copying files before that) and I’d already ruled cacheing out as a solution.

    > also, is he doing deadline on the guest, host, or both?

    deadline on the host – didn’t try it on the guest.

    > not sure if they implemented it yet, but they were talking about a vm host setting for tuned
    > and one for guests
    > yeah, Wagner mentioned it in his summit presentation:
    > they should be available in rhel 6.3 according to his presentation

    Well, tuned-adm is a gift for part-time sysadmins like myself.

    Some of the guest disk write figures were close to the host’s & better than CentOS 5 after doing…

    yum install tuned tuned-adm profile virtual-host

    ..in the host and…

    yum install tuned tuned-adm profile virtual-guest

    …in the guest.

    Here are the new bonnie++ guest block write figures in MB/s. all using tuned-host and virtio, with & without tuned-guest. Not sure why there’s so much variation, but at least they’re all much better.

    45 tuned-host
    73 tuned-host
    50 tuned-host + tuned-guest
    37 tuned-host + tuned-guest

    > rhel/CentOS 6 probably went with a more conservative tuning option

    Certainly looks that way. It’s be interesting to know what & why.

    Before jimi_c provided the tuned-adm tip, I was hoping that running the VM off a block device might be the answer. i.e:
    qemu-img convert /media/vm027/hda.raw -O /dev/vm/vm031

    …but they are worse than running off a raw virtual disk file.

    16 tuned-host virtio
    20 tuned-host virtio
    27 tuned-host tuned-guest virtio
    24 tuned-host tuned-guest virtio

    I’m not convinced, maybe there are other factors at work. I’d investigate further if my plan A wasn’t back on track.

    The bonnie++ figures I gave before measuring host disk write performance were for the host *root* partition, not the LVM volumes that the guest VMs use. What if the problem is LVM, not KVM? So I did some timings comparing the root drive with LVM volumes, some with & some without tuned-host. ‘archive’ and ‘vmxxx’ are the lvm volume names: (Note:
    these timings are done in the host, not in a guest)

    69 /
    69 /
    70 /
    65 / + tuned-host
    64 / + tuned-host

    55 archive
    56 archive
    53 vm027
    48 vm027
    33 vm027 + tuned-host
    38 vm027 + tuned-host
    53 vm022 + tuned-host
    85 vm022 + tuned-host

    This indicates that there is a wide variation in performance between different LVM volumes, and all the LVM volumes are performing worse than the root. (It’s interesting that with tuned-host, the times seem to be mostly worse, but with greater deviation.) I repeated the above test on the CentOS 5 server (without tuned-host of course) and found the same thing – LVMs perform worse than root and vary widely:

    54 /
    50 /
    39 archive
    45 archive
    39 archive
    49 vm022
    34 vm027
    33 vm027

    A slight performance hit might be expected for LVM, but I though it was meant to be negligible? If the figures fell into 2 bands – good and bad
    – then I’d be looking for a specific problem like a sector alignment, but they don’t, and isn’t sector alignment meant to be fixed on CentOS
    6? The variation in performance indicates a problem of variable severity like fragmentation or the position on the physical disk – but I
    don’t think either of those are likely causes, because there’s only one file in each volume, and physical disk position shouldn’t have such a marked effect should it? Any other suggestions?

    Thanks, Julian

  • Actually the physical disk position can make a HUGE difference. I have a machine with a velociraptor 1GB drive which I have parititioned into two
    500GB partitions, the average speed of the first (on the outside cylinders of the drive) is far better than the second on the inside. Sorry I don’t have specific numbers now but the machine is in service so I can’t bash on it anymore. However, it was a difference of almost 50%.

    Modern drives don’t have a fixed number of sectors/track. Instead they try to keep a fixed density so there are more cylinders on the outer tracks than the inner. Thus per revolution more sectors go past the head for the tracks on the outside (low numbered tracks) verses those on the inside.

    Given that modern drives have interface speeds that are considerably faster than the actual media I/O speed, the media I/O rate is the limiting factor and thus where on the disk your data lies makes a big difference in I/O

    For those reasons I always try to put swap and I/O critical stuff, like swap, at the beginning of the drive and loath partitioning software that thinks it’s smarter and puts things where it wants.

  • >> physical disk position shouldn’t have such a marked effect should it?

    Nanook wrote:
    > Actually the physical disk position can make a HUGE difference.

    Thank you Nanook for your explanation. I think you’re right. It looks like the variation in performance is not due to LVM, but the position on the disk. If the volumes’ write performance figures are rearranged in the order of their position on the disk, they are roughly in line.

    The root disk is first, so it is best. Then there are no figures for the swap file. The archive volume is roughly half the remaining space, so I’d expect it’s performance to be closer to root. But archive is quite full, so maybe the bonnie++ test files went towards the end, near the other VM partitions. The last volume to be allocated has the worst performance. It all makes sense!

    So, when calculating the disk performance hit from using a VM, or the effectiveness of VM disk optimisations, the benchmark is the performance of the volume that hosts the corresponding virtual disk, not the root disk.

    And when there are many VMs, each with its own root disk, swap file, temp disk, there’s a challenge! Well, at least I can try to ensure that the busiest VM gets the first logical volume.

    All this shows the importance of basing decisions on evidence from performance tests on your own servers, not assumptions & other peoples’

    Thanks, Julian

  • I did a “study”, more like a quick experiment a few months back to determine best combination between Preallocation and Caching and posted my results here, http://itscblog.tamu.edu/improve-disk-io-performance-in-kvm/. It’s not very scientific as it was just done by timing the same kickstart CentOS install 4 times with varying parameters for Preallocation and Caching. I also only used qcow2 images, which do not perform as well as raw.

    – Trey