How To Load Balance Interrupts Of A NIC On The PCI-MSI-edge Driver In CentOS? :(

Home » CentOS » How To Load Balance Interrupts Of A NIC On The PCI-MSI-edge Driver In CentOS? :(
CentOS 9 Comments

Hello CentOS !

Iam trying to configure a Dual Core 1.6Ghz AMD E-350 machine to be able NIC (eth0) interrupts the most efficient way possible so that I can be able to handle the large number of random source packets per second. Currently running CentOS 6.4 2.6.32-358.6.2.el6.x86_64, i made some load testing with a random sourced syn flood to the NIC of this machine and found out according to top all the CPU interrupts are being sent to CPU0, thus saturating it and keeping the second core 100% idle because of this the amount of PPS handled is low (50k /sec). So.. below the output of /proc/interrupts, and also the output of the SMP affinity file which already contains “3” which I expect it tells the kernel to load balance the interrupts between the two cores and its not happening.

Questions:
a.) Could anyody please help me with a solution so that iam able to balance equally the interrupts on both cores?
b.) Is there anyway to actually assign more interrupts to this IRQ (25)
or is that something that the kernel discretly takes care of?

Thanks

Alex

[root@node2 ~]# cat /proc/irq/25/smp_affinity
3

Btw I also tried setting this value to “2” and “1” to get to choose which core it handles, and that did work but again the problem is I can only use one core at a time and not balance!

[root@node2 ~]# cat /proc/interrupts
CPU0 CPU1
0: 229 9838 IO-APIC-edge timer
1: 0 2 IO-APIC-edge i8042
8: 0 1 IO-APIC-edge rtc0
9: 0 0 IO-APIC-fasteoi acpi
12: 0 4 IO-APIC-edge i8042
14: 0 0 IO-APIC-edge pata_atiixp
15: 0 0 IO-APIC-edge pata_atiixp
16: 4 650 IO-APIC-fasteoi snd_hda_intel
17: 0 0 IO-APIC-fasteoi ehci_hcd:usb1, ehci_hcd:usb2, ehc i_hcd:usb3
18: 0 4 IO-APIC-fasteoi ohci_hcd:usb4, ohci_hcd:usb5, ohc i_hcd:usb6, ohci_hcd:usb7
19: 7 2754 IO-APIC-fasteoi ahci
24: 0 31 PCI-MSI-edge snd_hda_intel
25: 1 876343 PCI-MSI-edge eth0

9 thoughts on - How To Load Balance Interrupts Of A NIC On The PCI-MSI-edge Driver In CentOS? :(

  • afaik, IRQs from a single device have to be handled serially, you can’t have more than one active IRQ at the same time.

    so there’s really no point in distributing them across multiple CPU
    cores. let the other cores handle user space and other sorts of IOs.

    SYN flooding is a pathological condition thats not going to happen under regular workloads, how does your system handle actual network traffic when it approaches network saturation ?

  • John,

    We run a reverse proxy so our CPU need for that is very small, plus we get syn flooded often.. which is why we have the need to be able to load between cores, i think it is justified.

    Thanks Alex

  • John,

    In summary, its very easy to hit a low number of packets/second flood and have the server start looking legitimate ones. I find it hard to believe that one cannot benefit from load balancing the interrupts?

    Alex

  • Hello again John, I only have one network adapter. GigE.

    Perhaps anybody else can give me another suggestion in order to achieve this?

    Thanks Alex

  • IRQ handlers are not reentrant, as the hardware can’t deal with it.

    you can only have one CPU thread at a time in the IRQ handler for a specific device, so you might as well just handle one device with one CPU. if you have several ethernet adapters, then I’d hope the IRQs could be distributed

    one device interrupting both cores would cause a lot of trouble with hardware contention, or it would need to be all spinlocked which would make it worse, you’d be CPU bound in TWO cores in spinlocks half the time.

    but again, a Server adapter like a Intel Pro1000 or whatever, something like… http://www.amazon.com/Intel-1000-Dual-Server-Adapter/dp/B000BMZHX2

    will do much better than some $5 realtek junk.

  • Thanks John!,

    I understand now. I think there are some NICs that have different queues which are connected to the kernel in the form of different IRQs, and as thus can be load balanced nicely. Ive seen that on 10GigE NICs.. but iam wondering if by any chance you know if the one suggested can do such, or the name of the feature Iam looking for so I can browse myself?

    Alex

  • Thanks John,

    You have been most helpful.

    Do you happen to have any suggestion to the following:

    In my lab test with two identical Core I5 machines I created a syn flood on a gigabit switch: In the originating machine I was able to get 800k pps using one core only and that core was used at 75% with top reporting most of the consumption was at SYS, and SI almost nothing. At the recieving side the core assigned to the IRQ of the NIC was maxed with
    100% SI (syn cookies was disabled).

    Why is clearly so significantly expensive for the recieving side rather than the sending side to process a syn flood if they have identical hardware? Oh and btw the recieving side could only do 400k pps.

    Thanks!
    Alex

  • Am 21.05.2013 20:05, schrieb Alex Flex:

    Sending is fundamentally less work than receiving.

    The sender just puts whatever it wants to send on the line. The receiver has to recognize it, analyze it, find out whether it is the intended recipient, match it with what it received before, keep state, etc.

    In the case of a SYN flood, the sender exacerbates this on purpose, by reducing its own workload (exploiting the fact that it doesn’t really want to communicate) and maliciously increasing the receiver’s workload (forcing it to maintain enormous numbers of half-open connections).

    So it’s really quite unsurprising that a SYN flood puts less load on the sender than on the receiver.

LEAVE A COMMENT