Kernel 2.6.32-573.22.1.el6.x86_64, Higher Than Usual Load

Home » CentOS » Kernel 2.6.32-573.22.1.el6.x86_64, Higher Than Usual Load
CentOS 4 Comments

Hi,

After switching to the 2.6.32-573.22.1.el6.x86_64 kernel (CentOS 6) a few weeks ago I have noticed that my servers are now reporting a much higher idle load.

The servers are being monitored with Zabbix, and there is a clear difference between the older kernel 2.6.32-573.12.1 and the newer 2.6.32-573.12.1.

I can see the problem on both physical and virtual servers.

You can see the Load Average chart here from one of our servers:
http://imgur.com/U1eihMX

The server is pretty much idle all the time, but the load average increased from 0.001 to 0.300 when updated to the new kernel.

Is anyone else seeing the same behavior?

Best Regards, Johnny Carlsen

4 thoughts on - Kernel 2.6.32-573.22.1.el6.x86_64, Higher Than Usual Load

  • –ehHDV0GBaKqdUGcu2baqXucEv8GvGehbN
    Content-Type: text/plain; charset=windows-1252
    Content-Transfer-Encoding: quoted-printable

    This is one of the fixes listed on the errata page:

    “Due to prematurely decremented calc_load_task, the calculated load average was off by up to the number of CPUs in the machine. As a consequence, job scheduling worked improperly causing a drop in the system performance. This update keeps the delta of the CPU going into NO_HZ idle separately, and folds the pending idle delta into the global active countwhile correctly aging the averages for the idle-duration when leaving NO_HZ mode. Now, job scheduling works correctly, ensuring balanced CPU load. (BZ#1300349)”

    From:
    https://rhn.redhat.com/errata/RHSA-2016-0494.html

    No idea if that is impacting the load shown on your machine .. maybe someone has some ideas.

    –ehHDV0GBaKqdUGcu2baqXucEv8GvGehbN

  • This is one of the fixes listed on the errata page:

    “Due to prematurely decremented calc_load_task, the calculated load
    (snip)”

    From:
    https://rhn.redhat.com/errata/RHSA-2016-0494.html

    No idea if that is impacting the load shown on your machine .. maybe someone has some ideas.

    An RHEL user is reporting what seems to be the same issue:

    https://access.redhat.com/discussions/2247501

    Thanks, it is nice to see that others are experiencing this too and that it is already reported upstream.

    Regards, Johnny

  • I did a batch update on a number of Xen vservers, with kernel 2.6.32-573.26.1 being installed.

    In addition to load averages being high and these #s not being reflected by the individual CPU numbers within top, I now have a spattering of occasional daemon crashes that the vservers have historically never had. Such as, here is the start of a nameserver blowing up in my logs:

    May 12 04:03:41 XXX kernel: named invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 : #

    May 12 04:03:41 XXX kernel: named cpuset=/ mems_allowed=0 : #

    May 12 04:03:41 XXX kernel: Pid: 2208, comm: named Not tainted 2.6.32-573.26.1.el6.x86_64 #1 : #

    May 12 04:03:41 XXX kernel: Call Trace: : #

    This nameserver has never had a problem with its memory. And, there is another daemon on a different vserver that has done it twice, with the same sort of OOM thing.

    And.. what kind of sense does this make.

    top – 17:48:53 up 2 days, 9:10, 2 users, load average: 0.47, 0.29, 0.24
    Tasks: 95 total, 1 running, 93 sleeping, 0 stopped, 1 zombie Cpu0 : 0.3%us, 0.7%sy, 0.0%ni, 96.7%id, 2.3%wa, 0.0%hi, 0.0%si, 0.0%st Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

    There’s some problem here, at least because of the crashing out of nowhere on formerly-fine daemons. I’ll be reverting all my kernels back to 2.6.32-573.18.1.