Increase CPU Usage On HV After Upgrade (7.2 -> 7.3)
Hello ,
After upgrading the system from CentOS 7.2.1511 to CenOS 7.3.1611 I
see that the average processing time has increased from 5-7% to 12-15%
(doubled). Not critical but it is not pleasant. Server as KVM with 5
virtual machines. Someone noticed something similar? If so, how to fix that?
Thx.
14 thoughts on - Increase CPU Usage On HV After Upgrade (7.2 -> 7.3)
In the mean time, if you have not disabled it, you should find some collected statistics from sysstat/sar. Look at the sarXX files under /var/log/sa. They should be kept for 30 day by default in CentOS 7. So you can compare cpu, mem, I/O profiles before and after the upgrade. If you have access to Red Hat documents you can look also here:
https://access.redhat.com/articles/325783
or in general some articles like this:
https://www.blackmoreops.com/2014/06/18/sysstat-sar-examples-usage/
and to create pdf graphics with kSar https://www.thomas-krenn.com/en/wiki/Linux_Performance_Analysis_using_kSar
HIH, Gianluca
How is your storage arranged, and what kind of IO patterns do those VMs have?
During recent testing, I found that the read performance of software RAID volumes was worse under 7.3 than it was under 7.2. Most other IO
had improved significantly:
https://plus.google.com/+GordonMessmer/posts/eSe6iNmk1Fs?sfc
Discussed with KVM guys, here’s the answer:
1% per VM seems to much on idle VM; if the VM is not idle you are getting better performance in exchange. For idle VMs the difference should be roughly 0.1% per VM.
Hello Gordon,
Wednesday, January 18, 2017, 11:52:35 PM, you wrote:
It is software RAID1 + LVM
Do not quite understand. What do you mean?
Hello Gianluca,
Wednesday, January 18, 2017, 3:54:15 PM, you wrote:
Unfortunately, on that host such statistics is disabled.
Hello Subscriber,
Thursday, January 19, 2017, 4:44:04 PM, you wrote:
But I collect such statistics in Zabbix. And the numbers and graphs indicate an increase in the load on the CPU (ie System time).
What at the VMs doing? Are they entirely idle? Are they doing light work, mostly reading from disks? If they’re not generating disk IO, then that’s not related. However, during a recent set of benchmarks, I
found that disk reads were slower under 7.3 than under 7.2. That might be specific to the system I tested, or it might be related to the change you’re seeing.
“load” has another meaning in the context of POSIX system performance counters. I’m pretty sure you’re talking about CPU utilization and not
“load”, right?
Hello Gordon,
Thursday, January 19, 2017, 5:09:29 PM, you wrote:
Definitely.
Subscriber wrote:
That’s surprising. That’s such an old, low-level daemon/reporting tool…. Was it disabled deliberately? And why?
mark
Hello Gordon,
Thursday, January 19, 2017, 4:57:48 PM, you wrote:
Its gateway from local network to Internet
At work time – No. Another time in most – Yes
Well no. They are not loaded the disc(s). No heavy for write operations.
Hello m,
Thursday, January 19, 2017, 5:17:48 PM, you wrote:
Heritage from the old admin to me.
Hello Sandro,
Friday, January 20, 2017, 6:54:02 PM, you wrote:
kvm_intel
In CentOS 7.2.1511:
kernel-3.10.0-327.36.3.el7.x86_64
qemu-kvm-1.5.3-105.el7_2.7.x86_64
libvirt-daemon-kvm-1.2.17-13.el7_2.5.x86_64
After upgrade from 7.2.1511 to 7.3.1611:
It was (after upgrading to this version, I noticed described in the first report the situation):
kernel-3.10.0-514.2.2.el7.x86_64
qemu-kvm-1.5.3-126.el7.x86_64
libvirt-daemon-kvm-2.0.0-10.el7_3.2.x86_64
Now (as of 2017/01/22):
kernel-3.10.0-514.6.1.el7.x86_64
qemu-kvm-1.5.3-126.el7_3.3.x86_64
libvirt-daemon-kvm-2.0.0-10.el7_3.4.x86_64
It takes 1-2 days for monitoring, to determine whether the situation has changed since the last update.
It is hard to define that extra % are leak due to upgrade, maybe should try on little heavy load VM and see if it is still constant.
Xlord
From: CentOS-virt [mailto:CentOS-virt-bounces@CentOS.org] Hello ,
After upgrading the system from CentOS 7.2.1511 to CenOS 7.3.1611 I
see that the average processing time has increased from 5-7% to 12-15%
(doubled). Not critical but it is not pleasant. Server as KVM with 5
virtual machines. Someone noticed something similar? If so, how to fix that?
Discussed with KVM guys, here’s the answer:
1% per VM seems to much on idle VM; if the VM is not idle you are getting better performance in exchange. For idle VMs the difference should be roughly 0.1% per VM.
Thx.