Tcpdump Loses Lots Of Packets

Home » CentOS » Tcpdump Loses Lots Of Packets
CentOS 5 Comments

Hi folks,

I’ve got a Dell R320 running CentOS 7, and a 10G NIC. I’m running a DNS
server on it, for testing. As part of my testing, I’m attempting to capture all the DNS queries arriving on the server, using tcpdump. However, tcpdump’s performance is abysmal, and it loses lots of the packets. Here’s the output when I stop the capture:

# tcpdump -nn -i p1p1 -w /dev/shm/queries.pcap -B 65536 dst host
tcpdump: listening on p1p1, link-type EN10MB (Ethernet), capture size
65535 bytes
^C4096558 packets captured
6882162 packets received by filter
2785604 packets dropped by kernel

The number 6882162 is exactly the number of DNS queries I am sending from another server (the source). The filter is seeing them. However, not all of them make it into the pcap file. First I was writing to the disk, and thought it might be a bottleneck, so I tried to write to
/dev/shm. You’ll see I’ve also increased the buffer size with -B, but none of this helps.

The packet rate is also not that high. From the sending side, this is what I have:

# tcpreplay -i qtx:p1p1 5min.pcap Actual: 6882162 packets (620533334 bytes) sent in 317.08 seconds. Rated: 1951900.0 Bps, 15.61 Mbps, 21648.97 pps Flows: 4254416 flows, 13382.96 fps, 6882162 flow packets, 0 non-flow

This is a very modest rate.

Any ideas why tcpdump loses so many packets? And how to make it do better?

Regards, Anand

5 thoughts on - Tcpdump Loses Lots Of Packets

  • Try specifying “ether host ” and compare the pcap files. How are you counting the number of packets in the pcap file?

  • Capturing 10 gigabit traffic with no packet loss at line speed is difficult at best. Make sure that you’ve configured the IRQ affinity properly on the sending and receiving sides to start.

  • Hi folks,

    I’ve discovered something. See below:

    If I send packets without qtx, like this:

    tcpreplay -i p1p1 5min.pcap

    then tcpdump on the receiving box has no problem, and keeps up happily with the queries, and receives all of them into the pcap file. It seems like the qtx module is somehow interfering with the packet capture, but I don’t know how or why yet.

    Anyway, for low packet rates, such as 20,000 q/s, qtx isn’t necessary, so I will not use it when I want to do packet captures.

    Regards, Anand

  • Hello,

    I’ve found it is helpful to limit the length of the packet you are capturing by using something like -s 256.

  • Saw your nanog posts…

    How many RX queues are configured? What does ‘ethtool -S p1p1’ show? Any discarded packets in the RX queue(s)?