CentOS7, Xen 4.6.1, Kernel 3.18.25-19 Strange Performance Problem

Home » CentOS-Virt » CentOS7, Xen 4.6.1, Kernel 3.18.25-19 Strange Performance Problem
CentOS-Virt 2 Comments

Hi all,

i just stumbled over a strange performance issue with my Xen setup.

I use CentOS-virt Xen since a long time on my workstation and usually never check performance. However, yesterday I booted into the 3.18.25-19
Dom0 kernel *without* Xen and found the system noticably more responsive. That triggered me into running a simple kernel compile benchmark and compare the results:

Testsequence:
echo 3 > /proc/sys/vm/drop_caches make distclean cp ../.config . #copies an existing valid config for compile make oldconfig time make -j bzImage

Results:

xen 4.6.1 – kernel 3.18.25-19
16 threads:
real 1m41.983s user 16m37.247s sys 5m7.121s

32 threads:
real 1m29.240s user 23m7.170s sys 6m31.497s

kernel 3.18.25-19
16 threads:
real 1m12.909s user 14m37.627s sys 1m36.521s

32 threads:
real 1m0.530s user 18m45.569s sys 1m58.807s

Some notes:
– System is a HP z840 with 2*8core E5 v3 Xeon and 128gig Ram. Disk is an Intel 750 ssd (PCIe nvme)
– While normally in Xen, I limit Dom0 Memory via dom0_mem, but for the tests, Dom0 was allowed to use full memory
– While running the test, no guest was running except Dom0. System was booted to graphical desktop and was completely unloaded otherwise
– (boot-)logs showed no issues with ACPI or hardware-support or whatsoever.

I have not yet run any in-depth investigation or tried too many different settings, but I find the differences quite noteworthy. So, if anybody came across the same issues, has some hints or suggestions, feel free. I’ll continue poking around and if there is interest, i’ll post updates.

Regards, Thomas

2 thoughts on - CentOS7, Xen 4.6.1, Kernel 3.18.25-19 Strange Performance Problem

  • Thanks for the report. 64-bit PV is known to be slow for Xen for workloads that involve a lot of system calls (which kernel build certainly will). You can read an explanation for the issue, and our intended solution, in this blog post I wrote a few years ago:

    https://blog.xenproject.org/2012/10/31/the-paravirtualization-spectrum-part-2-from-poles-to-a-spectrum/

    Unfortunately the initial implementation of PVH turned out to have some technical issues, and it’s in the process of being revamped. The work is actively ongoing, but it’s likely to be another year at least before we’re going to be able to default to a PVH dom0 kernel for CentOS.

    Still, over 3x for a kernel build is rather surprising — I’ll ask around and see what other people’s expectations / experiences are. It’s possible there’s a regression in the CentOS build that hasn’t been noticed. In particular, if you happen to find any other combination of Xen / kernel which is significantly faster, do report it so we can look into it.

    -George

  • Right, thanks for the link. I am aware, that the PV-part of PVHVM is eating some performance. Usually, this is something in the 10-20%
    ballpark, give or take the workload. But this is more a 50% hit in walltime and close to a 5-fold jump in system time. Also keep in mind, that this is an E5 v3 based machine, with APIC-V (and Xen detecting it). I was expecting:
    – that Dom0 utilizes it, however I cannot confirm it’s usage
    – that this will at least help with irq processing times. I am aware of PVH (and it’s potential merits). However, as it is not there yet, I have no reference as to what to expect in real life. The point I am making, is that the performance hit observed is quite hefty, IMHO. I have never observed something in that magnitude before.

    Note, that I might be looking at a problem, entirely unrelated to Xen. Yes, that is precisely what I was hoping for – other people’s expectations/experiences. Maybe even some repeats of my simple test and posted results. On that machine, I use Xen as a workstation setup, where Dom0 also serves as my regular work environment. On the infrastructure, I maintain at work, Dom0 performance is next to irrelevant. I have tried some additional things. The whole thing started with a
    4.6.0-rc2 kernel tried as Dom0 kernel, as I wanted to try the new multiqueue blkback/blkfront patches. Unfortunately, that did not work due to my graphics not being initialized properly, which I assume is unrelated to Xen. I’ll continue poking around. I have tried some non-Xen bare-metal kernels, including said 4.6.0-rc2, which yielded results consistent with the 3.18.29 being used as bare-metal kernel. I even applied a BIOS-update, which did not change things either, just to make sure (according to my experience, the z-series Workstations are pretty good at delivering working BIOSes, which is one of the main reasons for me to buy them in the first place).

    I’ll continue investigating and post updates, if i find anything noteworthy. I’d be more than happy to hear, what kinds of results others see and i am open to suggestions for things to try.

    Regards, Thomas