I have 2 similar servers. Since upgrading one from CentOS 5.5 to 6, disk write performance in kvm guest VMs is much worse.
There are many, many posts about optimising kvm, many mentioning disk performance in CentOS 5 vs 6. I’ve tried various changes to speed up write performance, but northing’s made a significant difference so far:
- Install virtio disk drivers in guest
- update the host software
- Update RAID firmware to latest version
- Switch the host disk scheduler to deadline
- Increase host RAM from 8GB to 24GB
- Increase guest RAM from 2GB to 4GB
- Try different kvm cache options
- Switch host from ext4 back to ext3
- Set noatime on the virtual disk image file Note: There is no encryption or on-access virus scanner on any host or guest.
Below are some the block write figures in MB/s from bonnie++ with various configurations:
First, figures for the hosts show that the CentOS 6 server is faster:
54 CentOS 5 Host
50 CentOS 5 Host
69 CentOS 6 host
70 CentOS 6 host
Figures for a CentOS 6 guest running on the CentOS 5 host show that the performance hit is less than 50%:
30 CentOS 6 guest on CentOS 5 host with no optimisations
27 CentOS 6 guest on CentOS 5 host with no optimisations
32 CentOS 6 guest on CentOS 5 host with no optimisations
Here are the figures a CentOS 6 guest running on the CentOS 6 host with various optimisations. Even with these optimisations, performance doesn’t come close to the un-optimised guest running on the CentoOS 5 host:
5 No optimisations (i.e. same configuration as on CentOS 5)
4 deadline scheduler
5 deadline scheduler
15 noatime + deadline scheduler
10 virtio + noatime
9 virtio + noatime
The CentOS 6 server has a better RAID card, different disks and more RAM, which might account for the better CentOS 6 host performance. But why might the guest write performance be so much worse?
Is this a known problem? If so, what’s the cause? If not, is there a way to locate the problem rather than using trial and error?