Very Slow Disk I/O

Home » CentOS » Very Slow Disk I/O
CentOS 25 Comments

Hi Users

I am using RHEL 6.5 on my server.

From top command i can see that the processors in my server are spending a lot of time on wait for I/O. I can see high percentage in terms of 30-50% on “wa” time.

Here is the df output about the disk space in my system:

**********
[root@localhost images]# df Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 1915071844 103227908 1714563936 6% /
tmpfs 32931472 0 32931472 0% /dev/shm
**********

Could someone point me on how to improve the disk I/O on my server and reduce the wait time on I/O.

Thanks Jatin

25 thoughts on - Very Slow Disk I/O

  • +1

    And remember that I/O is more than just disk. The atop monitor gives you information like top and htop, but also provides a lot of I/O information as well including network. Perhaps your server is the target of a network-based DDOS attack which can cause lots of I/O wait time. Also look at top to see what numbers are in the si and hi columns. THese stand for software and hardware interrupts. If one of those is high you can also narrow it down.

    Hope this helps.

  • it appears you have everything on a single file system on a single 2TB
    drive? I’m going to guess thats a desktop grade SATA disk spinning at most at 7200rpm (many are slower, ‘green’ disks are often 5400rpm).

    what sort of workload are you running? our high disk IO servers, doing things like transactional databases, have many 10000rpm or 15000
    rpm drives on a SAS2 raid card with 1GB or 2GB of write-back-cache
    (battery or flash backed).

  • oops, left off the last part. To improve disk IO performance, use more/faster disks, or do less disk IO. Since you’ve given us no clues about your hardware configuration OR software workload, its hard to be more specific.

  • Start by identifying your disk and controller. Assuming that this is a single SATA disk:

    # smartctl -a /dev/sda | egrep ‘Model:|Rate:|SATA Version’
    # lspci | grep SATA

    Next install and run iotop. If there’s something on your system that’s particularly active and unnecessary, disable it.

    If everything contending for your disk *is* necessary, then you may need to take steps to improve disk performance. There are a number of possibilities.

    1: Use an SSD instead of a disk.

    2: Use an SSD in addition to a disk. If a purely SSD storage system is too expensive, you might choose to use an SSD for one or more filesytems that you can mount where the high volume IO is occurring, and continue to use the disk for filesystems that need more data capacity and fewer IOPS.

    3: Use a fast RAID array. A RAID1 array can provide better read performance than a single disk. For higher write volumes, use RAID10 on four or more disks. Avoid RAID 5 or 6, as they’ll be slower than a single disk.

    4: Replace poorly supported hardware. It might be that your SATA
    controller isn’t well supported, and performs badly as a result. I used to have a desktop system with NVidia motherboard chipset. Its SATA
    controller was garbage. Performance with the same disk was noticeably better with a cheap PCI-E SATA controller.

  • [Jatin]
    [root@localhost images]# smartctl -a /dev/sda | egrep ‘Model:|Rate:|SATA
    Version’
    [root@localhost images]#

    [Jatin]
    [root@localhost images]# lspci | grep SATA
    00:11.4 SATA controller: Intel Corporation Wellsburg sSATA Controller
    [AHCI mode] (rev 05)
    00:1f.2 SATA controller: Intel Corporation Wellsburg 6-Port SATA
    Controller [AHCI mode] (rev 05)
    [Jatin]
    Thanks for these inputs. I will surely take it up with my team to see if we can incorporate new hardware for our product.
    [Jatin]
    We are using RAID 10 in our server with 4 disks.

    Hi Gordon

    Please see my responses inline to your questions. Appreciate your inputs in this direction.

    Thanks Jatin

  • [Jatin]
    Yes , entire disk is mounted on a single file system. Will this cause high wait time for I/O operations ? Do you think splitting it into multiple partitions would help ? Appreciate if you can enlighten me in this regard or provide pointers to it.
    [Jatin]
    Yes , it is a SATA disk. I am not sure of the speed. Can you tell me how to find out this information ? Additionally we are using RAID 10
    configuration with 4 disks.
    [Jatin]
    Our workload is a bit high on disk reads / writes and hence i am seeing slowness in it when using the top command. We basically run VMs on the RHEL 6.5 host. Our target is to house 5-6 application VMs on this host and ship this as a product on a rack server. Hi John

    Please see my responses inline. Appreciate your inputs in this regard.

    Thanks Jatin

  • The extra time goes to move the head back and forth from one file to the next as different applications write to different files. Different partitions may make things even worse if they happen to force those locations to be farther apart. What you need are more drives with activity spread over them so you have a better chance of a head being near where it needs to be. Adding RAM can help to a certain extent by allowing more buffering. Faster disks will help too, but won’t magically eliminate seek time – moving heads is still going to be your slowest computer operation.

    You can micro-manage this by putting a VM image on its own drive so you don’t lose seek time between the images. Or make a raid with a very large number of members and hope the odds work out.

  • [Jatin]
    [root@localhost ~]# lspci | grep RAID
    05:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS-3
    3108 [Invader] (rev 02)

  • to get info out of those, you need to install MegaCli64 from LSI Logic, which has the ugliest command lines and output you’ve ever seen.

    I use the python script below, which I put in /root/bin/lsi-raidinfo and chmod +x it, then run it like this..

    # lsi-raidinfo
    — Controllers

  • I’d expect some pretty good numbers from that with a 4 disk RAID 10. As before, SSD will perform better. If you need 2TB of space at similar cost, maybe consider 8 500GB drives instead of 4 1TB drives.

    Since your workload is in VMs, you’ll also see a large (potentially very large) performance benefit from a change to your filesystem layout. I
    recommend something closer to the default: one partition for /boot, and a second partition spanning the rest of the disk (RAID volume). The second partition should be part of an LVM group, your root filesystem should be limited to what it needs. Maybe a 40GB root FS. Use LVM
    backed VMs instead of file backed VMs.

    When you use file backed VMs, all of your disk IO goes through the filesystem layer, and possibly through the journal, which amplifies your write count. That’s costly.

    Beyond that, you might just want to test the system and make sure that it’s performing as expected. Figure out how much RAM is on your RAID
    controller and how much is available to the host system. Shut down all of your VMs and run “bonnie++” with a data size at least 2x the size of system RAM + RAID controller RAM.

  • Thanks John,

    I ran your script and here is the output for it:

    Start of the Output***************************
    [root@localhost bin]# lsi-raidinfo sh: /opt/MegaRAID/MegaCli/MegaCli64: No such file or directory
    — Controllers

  • Hi John

    I managed to install the software from LSI on my box and ran the script that you had provided and here is what i get from it.

    ************Output ******************
    [root@localhost bin]# lsi-raidinfo
    — Controllers

  • so, you have 2×2 ST91000640NS
    http://www.seagate.com/products/enterprise-servers-storage/nearline-storage/enterprise-capacity-2-5-hdd/

    those are “Nearline” disks, 7200RPM. They are intended for bulk secondary storage, archives, backups and such.

    you said you have a number of virtual machines all attempting to access this raid10 at once? I’m not surprised that it is slow. You’re probably limited by random IO/second, that raid likely does around 250 random operations/second. share that between 6-7 virtual systems and if they are all doing disk IO, they are going to slow down each other.

  • So , You dont think that any configuration changes like increasing the number of volumes or anything else will help in reducing the I/O wait time ?

    Thanks Jatin

  • not by much. it might reduce the overhead if you use LVM volumes for virtual disks instead of using files, but if you’re doing too much disk IO, there’s not much that helps other than faster disks (or reducing the amount of reads by more aggressive caching via having more memory).

  • No, because that won’t change the number of heads that are present to service the IO requests, nor to segregate requests effectively. Your primary goals should be to reduce IO (investigate using LVM backed VMs instead of file-backed) or to increase hardware resources (possibly a larger number of smaller disks if SSDs are not in budget).

  • Thanks John

    I will test and get the I/O speed results with the following and see what works best with the given workload:

    Create 5 volumes each with 150 GB in size for the 5 VMs that i will be running on the server Create 1 volume with 600GB in size for the 5 VMs that i will be running on the server Try with LVM volumes instead of files

    Will test and compare the I/O responsiveness in all cases and go with the one which is acceptable.

    Appreciate your responses in this regard. Thanks again..

    Regards, Jatin

  • Unless you put each VM on its own physical disk or raid1 mirror you aren’t really doing anything to isolate the vms from each other or to increase the odds that a head will be near the place the next access needs it to be.

  • [Jatin] Sure , I have no idea about LVM so i will do my learning on it. Thanks for pointing to it.
    [Jatin] This is also one option that i can try for sure since usage of disks instead of SSDs certainly falls within my budget.

  • Lol – spinning disks? Really?

    SSD is down to like 50cents a gig. And they have 1TB disks… slow disks = you get what you deserve… welcome to 2015. Autolacing shoes, self drying jackets, hoverboards – oh, yeah, and 110k IOPS 1TB SamSung Pro 850 SSD Drives for $449 on NewEgg.

    dumbass

    —–Original Message—

  • Am 03.02.2015 um 10:14 schrieb Joseph L. Brunner:

    Right, *consumer grade* SSD prices were coming down thzat much. But I am sure the appliance Jatin is building up (while he uses the @cisco.com mail domain) is intended for enterprise usage and has to have enterprise grade hardware components. Just like he used an enterprise grade HDD
    (though a SATA and not a SAS model with only 7.2krpm which is, as John pointed out, not a good choice for a virtualization host) to build the machine.

    Investigate about SSDs made to be used in servers, SLC and not MLC type of chips, SAS interface. Then lets speak again after you have checked their prices.

    Regards

    Alexander