Strange Performance Issue On CentOS 6.7 Server

Home » CentOS » Strange Performance Issue On CentOS 6.7 Server
CentOS 7 Comments

I’m running CentOS 6.7 on my build servers, and on one of the servers the builds are taking almost an order of magnitude longer than usual. There are no runaway processes and there is plenty of free memory. So I suspected that file I/O might be slow, and sure enough, that appears to be the case. I ran a simple dd test and compared the results to a “normal” build server (412 MB/s vs. 31.7 MB/s). Both are on similarly configured VGs on identical hardware. What could cause this performance degradation and are there any other tests I can run before I reboot the server to see if things improve? There is nothing in /var/log messages – are there other logs I should check?

Alfred

7 thoughts on - Strange Performance Issue On CentOS 6.7 Server

  • A dying hard disk can do it. HDDs try to silently paper over I/O errors, but what they can’t hide is the time it takes to do this. If your HDD is constantly correcting errors at the oxide layer, it will be reeeeeallly sllllow.

    You can try running SMART tests on it, though that’s not guaranteed to show the problem.

    Got tested backups? :)

  • Well, it’s not “a” disk: it’s a HW RAID of about dozen (server grade) drives, with a VG/LV on top of that. Are there any log files I can check that test the underlying VG/LV health status?

    Alfred

  • Alfred von Campe wrote:
    HP-based ones, both have utilities to check out the drives (MegaRAID and hpacli, respectively); AC&NC JetStors have a web interface.

    mark

  • smartctl can see through several different types of RAID controller to the underlying physical disks via its -d option.

  • This is what I have:

    # smartctl –all /dev/sda smartctl 5.43 2012-06-30 r3573 [i686-linux-2.6.32-573.12.1.el6.i686] (local build)
    Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

    Vendor: IBM
    Product: ServeRAID M5110e Revision: 3.19
    User Capacity: 1,494,996,746,240 bytes [1.49 TB]
    Logical block size: 512 bytes Logical Unit id: 0x60050760408e81b018be601809efd11c Serial number: 001cd1ef091860be18b0818e40600705
    Device type: disk Local Time is: Wed Feb 3 17:13:34 2016 EST
    Device does not support SMART

    Error Counter logging not supported Device does not support Self Test logging

    I guess I am stuck since it says it doesn’t support SMART. Or is there some way to get some status from this “disk” to see if it’s really the root cause of my performance issues. I think I would have seen something in /var/log/messages if there was a critical issue.

    Alfred

  • the underlying physical disks via its -d option.
    (local build)
    some way to get some status from this “disk” to see if it’s really the root cause of my performance issues. I think I would have seen something in
    /var/log/messages if there was a critical issue. Severely degraded hardware RAID performance can often be caused by things like a failed cache battery.

    There is usually some sort of tool to interrogate the device to check things like cache behaviour.

  • A bit of Googling says that’s an LSI 2208 based card.

    So, try smartctl -a -d megaraid,0

    If that works, you should be able to walk through each disk by incrementing that trailing number. Then, you can add -t flags to do active tests.