bitrate is too slow with tcpreplay v3.3.0 or newer

[awzilkie]

I am running tcpreplay on Linux FC5 to replay a captured MPEG audio stream. When I use tcpreplay version 3.2.5 or older, the stream plays at 38.27 pps (0.28 Mbps). But if I use tcpreplay version 3.3.0 or newer the stream plays slower (e.g. 31.83 pps/0.24 Mbps). When playing the stream through our audio card, this translates to choppy audio.

[aturner]

Try a different timing method. http://tcpreplay.synfin.net/wiki/tcpreplay#ChoosingaTimingMethod

[awzilkie]

Replying to aturner:

Try a different timing method. http://tcpreplay.synfin.net/wiki/tcpreplay#ChoosingaTimingMethod

I tried replaying the same capture with both versions of tcpreplay using the same timing method (nano). v3.2.5 runs faster than v3.4.3. I tried other timing methods (rdtsc and gtod) with v3.4.3 just to be sure, but the rate was still to slow.

[aturner]

please attach sample pcap so I can look into this ok?

[awzilkie]

Sorry, attached the wrong file, how do I remove it?

[awzilkie]

Done. Had to make it less than 1 MB in order for it to upload. Thanks for looking into this

[aturner]

I'm having a hard time reproducing your problem on v3.4.3:

[root@monster tcpreplay-3.4.3]# ./src/tcpreplay -i eth0 -T nano /root/audio.pcap 
sending out eth0 
processing file: /root/audio.pcap
Actual: 568 packets (553185 bytes) sent in 15.66 seconds
Rated: 35324.7 bps, 0.27 Mbps, 36.27 pps
Statistics for network device: eth0
        Attempted packets:         568
        Successful packets:        568
        Failed packets:            0
        Retried packets (ENOBUFS): 0
        Retried packets (EAGAIN):  0
[root@monster tcpreplay-3.4.3]# ./src/tcpreplay -i eth0 -T gtod /root/audio.pcap 
sending out eth0 
processing file: /root/audio.pcap
Actual: 568 packets (553185 bytes) sent in 14.80 seconds
Rated: 37377.4 bps, 0.29 Mbps, 38.38 pps
Statistics for network device: eth0
        Attempted packets:         568
        Successful packets:        568
        Failed packets:            0
        Retried packets (ENOBUFS): 0
        Retried packets (EAGAIN):  0

Could you give me more information about your system? Kernel, hardware (CPU, NIC, motherboard chipset, etc) and anything else you think might be relevant?

[awzilkie]

Before I give you more information there is one thing that I have observed with using -T gtod that seems suspicious:

if I run tcpreplay-3.4.3/src/tcpreplay -i eth0 -T gtod /tmp/audio.cap.rewrite &

and strace it, all I see is many gtod() calls in between the send() calls (no sleeps). It would seem that gtod() is being called in a loop until the next packet time arrives. Should this be happening? Of course this will take up 100% of the CPU... This results in tcpreplay -T gtod being dependant on CPU speed and usage. Indeed if I do activity on my system (e.g. scrolling my konsole window) it affects the pps.

[awzilkie]

Here is some system information:

[root@Adam ~]# uname -a Linux Adam 2.6.15-1.2054_FC5 #1 Tue Mar 14 15:48:33 EST 2006 i686 i686 i386 GNU/Linux [root@Adam ~]# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel? cpu family : 15 model : 2 model name : Intel(R) Pentium(R) 4 CPU 2.80GHz stepping : 9 cpu MHz : 2793.463 cache size : 512 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid xtpr bogomips : 5595.03

[root@Adam ~]# cat /proc/meminfo MemTotal?: 255612 kB MemFree?: 12504 kB Buffers: 7968 kB Cached: 92800 kB SwapCached?: 66864 kB Active: 180208 kB Inactive: 38644 kB HighTotal?: 0 kB HighFree?: 0 kB LowTotal?: 255612 kB LowFree?: 12504 kB SwapTotal?: 524280 kB SwapFree?: 388628 kB Dirty: 4 kB Writeback: 0 kB Mapped: 155488 kB Slab: 15944 kB CommitLimit?: 652084 kB Committed_AS: 544732 kB PageTables?: 3600 kB VmallocTotal?: 770040 kB VmallocUsed?: 3004 kB VmallocChunk?: 766896 kB HugePages?_Total: 0 HugePages?_Free: 0 Hugepagesize: 4096 kB

---

eth0 NIC: Intel Corporation 82541PI Gigabit Ethernet Controller

[aturner]

What you describe regarding -T gtod is exactly how it works. Yes, it eats CPU, but gains accuracy. It is even more accurate if your kernel/CPU support the HPET timer and your version of glibc is new enough to use that for calls to gettimeofday().

As I explain in the wiki link I gave you in my first reply, generally speaking non-RTOS do a very poor job of "sleeping" for short periods. That's why -T nano doesn't always work as well if you need accurate timing. For longer times nanosleep does a better job, but it's still best effort.

You might want to try -T rdtsc too. The good news is you can tweak it via --rdtsc-clicks and --rdtsc-accel to compensate for timing issues. Bad news is that it's horribly unreliable on some systems if they implement certain CPU power saving tricks.

Anyways, I don't see anything wrong with your setup other then you're pretty short on RAM. The fact that you've swapping a fair bit (almost equal to the amount of RAM you have) is far from ideal... is this a VPS hosted somewhere? Performance always takes a hit under virtualization and getting consistent results always seems to be a challenge.

Anyways, it's unfortunate that under your specific situation that the default nanosleep method doesn't work as well as it used to under 3.2.5. 3.3.0 was a rewrite of the sleeping code to make it more efficient for high speeds and also I've fixed some strange corner case bugs. It's quite possible that an old bug was helping you or that your timing patterns work better in the old code. I'll look at the code some and see if I can massage it in a way so that nanosleep is as good as it used to be in the next release, but I can't make any promises.

[awzilkie]

FYI - using the 3.4.3 code with 3.2.5's version of do_sleep() works.

[aturner]

That would be expected... I'll look into trying to see what I can do without hurting high bit rate performance.

[aturner]

(In [2444]) small perf tweak refs #421