Frequently Asked Questions
How should Tcpreplay be capitalized?
When refering to the Tcpreplay suite of tools (tcpreplay, tcpprep, tcprewrite, etc) then the T is capitalized. When only refering to the tcpreplay standalone utility, then it's not. There is no such thing as 'TCPreplay' or 'TCP replay' as far as I'm concerned.
Is there a Microsoft Windows port?
Old Answer: Not really. We had one user port the code over for an old version of tcpreplay to Windows. Now we're looking for someone to help merge and maintain the code in to the main development tree. If you're interested in helping with this please contact Aaron Turner or the tcpreplay-users list. Other then that, you can download the tcpreplay-win32.zip file from the downloads section and give it a go. Please understand that the Win32 port of tcpreplay comes with no support whatsoever, so if you run into a problem you're on your own.
New Answer: As of 3.0.beta13, Windows 2000 and better are supported under Cygwin. My goal is to in the future natively support Windows, but I will most likely need some help with this. For more information, please read the Win32Readme.txt file.
How is Tcpreplay licensed?
Does tcpreplay support sending traffic to a server?
If by server you mean a daemon (Unix) or service (Windows) which listens on a port (a web or mail server would be common examples), then probably not. The biggest issue is that tcpreplay doesn't understand the state of common protocols like TCP. This means it is unable to synchronize Syn/Ack's to create valid TCP sessions.
In theory, ICMP and UDP based protocols should work fine if the MAC and IP addresses are correct. You can fix them using tcprewrite. However, some ICMP and UDP based protocols also carry Layer 3/4 protocol information in the protocol itself (SIP is one example), so if you change the IP addresses, it may not be a valid SIP packet anymore. In such cases, you may want to try using NetDude to edit the payload of the packets.
In the future, I hope to release flowreplay to allow you to replay pcap files to servers.
Tcpreplay doesn't send traffic as fast as I told it to!
Often people ask me why tcpreplay doesn't send traffic as fast as they told it to. Usually they're trying to do something like --mbps=100.0 (replay at 100Mbps) with a pcap of small packets. As it turns out, it's very hard for tcpreplay to hit faster speeds, especially if you're sending mostly small packets. Here are some tricks/ideas you can use to make tcpreplay send traffic as fast as you wanted:
- Make sure you're running the latest stable release of Tcpreplay. Older versions tend to have bugs in the timing code.
- Try a different timing mechanism. --timer=gtod or --timer=nano for example. OS X users should *always* use --timer=abstime
- Don't use --mbps, it's really slow. Use --pps instead.
- Use --pps-multi=X to cause tcpreplay send multiple packets each sleep cycle (only works with --pps).
- Use --preload-pcap to load the pcap file into RAM before sending.
- Use --sleep-accel=X to cause tcpreplay to sleep for less time then it calculated. Often using a small value of 2 or 3 is enough!
- Use --topspeed Always the fastest way to send packets!
- Lie! If you're trying to hit 100Mbps, tell tcpreplay to send at 115Mbps or some other higher value until it replays at the correct speed
- Make sure your network connection is fast enough. Don't try sending 50Mbps over a 10Mbps link for example.
Often using a combination of the above options will allow you to reach the speeds you're trying to hit.
Can I send packets on the same computer running tcpreplay?
Generally speaking no. When tcpreplay sends packets, it injects them between the TCP/IP stack of the system and the device driver of the network card. The result is the TCP/IP stack system running tcpreplay never sees the packets.
Older versions of tcpreplay allowed me to edit packets. What happened?
Version 2.x of tcpreplay allowed you to edit packets as they were being sent. This functionality was moved to tcprewrite and disabled by default in 3.0 due to performance reasons. The most recent versions of tcpreplay 3.x now ship two tcpreplay binaries: tcpreplay which is for best performance and tcpreplay-edit which allows editing while replaying at the same time.
Why do I need to use tcpprep to create cache files? Can't this be done in tcprewrite?
Most of the tcpprep modes take a holistic view of a pcap and looks at all the packets before deciding which IP's are servers and which are clients. This requires the pcap file to be processed fully, sometimes twice. This for all practicality excludes including the tcpprep logic in tcpreplay.
Secondly, since both tcpreplay and tcprewrite use tcpprep cache file data, it made sense to separate it into a standalone utility so that the processing can be leveraged across multiple rewrites and the actual packet sending.
Why is tcpreplay not sending all the packets?
Every now and then, someone emails the tcpreplay-users list, asking if there is a bug in tcpreplay which causes it not to send all the packets. This usually happens when the user uses the -t flag or is replaying a high-speed pcap file (> 50Mbps, although this number is dependant on the hardware in use).
The short version of the answer is: no, we are not aware of any bugs which might cause a few packets to not be sent.
The longer version goes something like this:
If you are running tcpreplay multiple times and are using tcpdump or other packet sniffer to count the number packets sent and are getting different numbers, it's not tcpreplay's fault. Try investigating:
- It is well known that tcpdump and other sniffers have a problem keeping up with high-speed traffic. Furthermore, the OS in many cases lies about how many packets were dropped. Tcpdump will repeat this lie to you. In other words, tcpdump isn't seeing all the packets. Usually this is a problem with the network card, driver or OS kernel which may or may not be fixable. Try another network card/driver.
- When tcpreplay sends a packet, it actually gets copied to a send buffer in the kernel. If this buffer is full, the kernel is supposed to tell tcpreplay that it didn't copy the packet to this buffer. If the kernel has a bug which squelches this error, tcpreplay will not keep trying to send the packet and will move on to the next one. Currently I am not aware of any OS kernels with this bug, but it is possible that it exists. If you find out that your OS has this problem, please let me know so I can list it here.
- Tcpreplay can't send packets which are larger then the MTU of the interface. Generally, you can increase the MTU beyond the default 1500 bytes by using the ifconfig utility.
- I've been informed by one user that Broadcom NIC's (I know some 10G cards, maybe others) have an RX checksumming feature builtin which causes it to drop packets, which causes tools like tcpdump miss packets even at very low rates (1Mbps). Disabling this feature caused the NIC to stop dropping packets.
Why are tcpreplay timings all messed up?
Occasionally someone complains about timings to be "messed up". Usually this seems to be caused by a pcap which contains packets with non-sensical timestamps. More specifically, I have seen cases where a packet has a timestamp before the previous packet in the capture file. I'm not sure how such a pcap got created, but it seems to occasionally happen.
There may be other reasons for this to happen, such as a bug in tcpreplay, so be sure you're running the latest version of code. If after checking these two possibilities, please contact the author so that he can help debug the issue.
Does tcpreplay support dual NIC's like Tomahawk?
Yes! tcpreplay has for many years supported high-speed playback using two interfaces (long before Tomahawk existed). For more information about how traffic is split between the two interfaces, see the tcpprep manpage.
Can tcpreplay read gzip/bzip2 compressed files?
Yes, but not directly. Since tcpreplay can read data via STDIN, you can decompress the file on the fly like this:
gzcat myfile.pcap.gz | tcpreplay -i eth0 -
Note that decompressing on the fly will require additional CPU time and will likely reduce the overall performance of tcpreplay.
How fast can tcpreplay send packets?
First, if performance is important to you, then upgrading to tcpreplay 3.x is worthwhile since it is more optimized then the 2.x series. After that, there are a number of variables which effect performance, including on how you measure it (packets/sec or bytes/sec).
Anyways, here are some sample test runs from an Intel P4 3.4Ghz using the Intel e1000/Pro network card with a Linux 2.6.x kernel:
First run (larger packets): tcpreplay -t -K -l 100000 -i eth2 -q --pktlen test/test.pcap Actual: 10000000 packets (7020500000 bytes) sent in 58.95 seconds Rated: 120843848.0 bps, 921.97 Mbps/sec, 172129.97 pps Statistics for network device: eth2 Attempted packets: 10000000 Successful packets: 10000000 Failed packets: 0 Retried packets: 0 Second run (smaller packets): tcpreplay -t -K -l 100000 -i eth2 -q test/test.pcap Actual: 10000000 packets (1062900000 bytes) sent in 16.63 seconds Rated: 63895116.0 bps, 487.48 Mbps/sec, 601139.50 pps Statistics for network device: eth2 Attempted packets: 10000000 Successful packets: 10000000 Failed packets: 0 Retried packets: 0
The key thing here is that when sending smaller packets (the 2nd run) the packets/second is much higher (almost 4x), but the overall Mbps is lower (about 1/2). Using a much more powerful system (dual quad-core Xeon 2.66Ghz) the first run was virtually identical while the smaller packets showed a noticeable improvement:
tcpreplay -t -K -l 100000 -i eth1 -q test/test.pcap Actual: 10000000 packets (1062900000 bytes) sent in 15.28 seconds Rated: 69537776.0 bps, 530.53 Mbps/sec, 654226.88 pps Statistics for network device: eth1 Attempted packets: 10000000 Successful packets: 10000000 Failed packets: 0 Retried packets: 0
Note Once you start limiting how quickly packets are sent (anything other then -t), there is a rather dramatic drop off in performance. I'm still working on how to create a low-overhead means of timing packets.
How can I make tcpreplay run faster?
Profiling tcpreplay has shown that a significant amount of time is spent writing packets to the network. Hence, your OS kernel implimentation of writing to raw sockets is one of the most important aspects since that is where tcpreplay spends most of it's time.
In no particular order:
- Use a fast enough hardware. tcpreplay is very dependant on disk I/O, your network card and CPU.
- When selecting disk drives for storing pcap files, choose drives with higher RPM over lower seek times or larger caches.
- SCSI disks with a good controller should provide the best performance, since they generally require the CPU to do less work.
- For best performance, consider some sort of striping RAID (like level 0 or 5) to increase read throughput performance.
- Don't store pcap files on a remote server (NFS or SMB mounted). That just increases latency and CPU utilization. Directly attached storage like SCSI or FibreChannel is fine.
- Since tcpreplay is not multi-threaded, SMP or dual-core processors won't help very much. However, other processes can run on the other CPU(s).
- Turn off hyperthreading (HT) if your CPU supports it.
- A good network card/driver is important. Realtek cards are known to be crap. Intel EEPro/1000 cards are cheap and work well.
- If you have enough RAM to store the pcap files in a RAM disk, that would be ideal.
- If you don't have enough RAM for a RAM disk, run tcpreplay against the file once in order to load parts of your file in your OS's disk cache. That will help improve performance the subsequent runs.
- If your computer is busy running a bunch of other processes (running X, downloading mail, etc) then it will impact tcpreplay's performance and ability to time packets correctly.
- Opening a bunch of small files repeatly will reduce performance. Consider using mergecap to generate a single large file.
- Like most network based I/O, it is faster to send the same amount of data in a few large packets then many small packets.
- If you are looping your pcap file, enabling the --enable-file-cache option should provide a significant performance boost assuming you have enough RAM.
Does tcpreplay support Endace DAG cards?
By default, tcpreplay does not support DAG cards. However, Endace has released a custom version of tcpreplay which does support their cards. Please note that the Tcpreplay developers do not support this custom version of tcpreplay, so if you have any questions, please contact Endace.
Can I use non-pcap capture files?
It turns out that there are quite a few capture file formats other then pcap. If you have a capture file created by a tool which uses one of these other formats (like Solaris snoop) you can convert it to pcap format by using Wireshark's tshark tool.
tshark -r blah.snoop -w blah.pcap
Does Tcpreplay support Pcap-Ng/NTAR files?
Yes. The Tcpreplay suite uses libpcap for reading & writing pcap files. If you have libpcap 1.1.0 or higher, then tcpreplay, tcprewrite, etc can read pcap-ng files. If you have an older version of libpcap, you should upgrade to the latest version as earlier versions of libpcap have bugs with pcap-ng files.
Can tcpreplay send packets over WiFi?
This turns out to be very OS/hardware dependent, but in many cases, the answer is yes. In order for things to work, you generally must do the following:
- Put the WiFi card in managed mode
- Your pcap files need to be DLT_EN10MB (Ethernet) and have a valid 803.2 header
- The source MAC of the packets need to match the MAC of your WiFi card
Why doesn't my application see packets replayed over loopback?
Most users are surprised when they try replaying UDP traffic over loopback that the listening daemon never sees the traffic. This turns out to be a limitation of the loopback interface on many operating systems. One contributing factor may be capturing traffic on an ethernet port, rewriting the IP addresses but not the L2 header. Since the loopback interface doesn't use an ethernet L2 header, the IP stack of the operating system is unable to parse the packet and deliver it to the listening daemon.
Can I use IPTables/Traffic Control with tcpreplay?
You can not use iptables/tc on the same box as you run tcpreplay. The only way to use IPTables or Traffic Control (tc) with tcpreplay is to run tcpreplay on a different box and send the traffic *through* the system running iptables/tc. This limitation is due to how the Linux kernel injects frames vs. reading frames for iptables/tc which makes traffic sent via tcpreplay to be invisible to iptables/tc.
Are there binaries available for XXX operating system?
Maybe. I do not release binaries for ANY operating system. Many operating systems like Linux, *BSD, Solaris and OS X have teams which package open source applications like tcpreplay and release them in their package format (RPM, BSD/Mac ports, SunFreeware, etc). Google is your friend.
What if I ask you really nicely to build a binary for me?
If by "nice" you mean you'll buy me some Boddingtons or a DVD off my Amazon wish list, yeah, I'll prolly help you out if I can. If by "nice" you mean saying "pretty please with sugar on top", then start Googling.
Unable to find a supported method to send packets. Please upgrade your libpcap or enable libnet
Tcpreplay can use a variety of API's/libraries to send packets: BSD's BPF, Linux's PF_PACKET, libpcap and libnet. If you're not running on a platform which supports BPF or PF_PACKET, you'll need either a recent version of libpcap or libnet. Honestly, between the two, I strongly suggest upgrading libpcap since it tends to have fewer bugs and is actively developed. Libnet by contrast has known bugs and hasn't been updated in a few years. For all intents and purposes, libnet support has been deprecated and will likely be removed in a future version.
Right now there is one case where libnet is necessary: you're not running on Linux or *BSD and you want to use tcpbridge. Right now there are limitations in the libpcap API which prevents tcpbridge from working when BPF and PF_PACKET aren't available.
Note: Tcpreplay no longer supports libnet!
tcpedit_stub.def: Command not found
Making all in tcpedit make: Entering directory `/home/acferen/tcpreplay-trunk/src/tcpedit' tcpedit_stub.def make: tcpedit_stub.def: Command not found make: *** [tcpedit_stub.h] Error 127 make: Leaving directory `/home/acferen/tcpreplay-trunk/src/tcpedit' make: *** [all-recursive] Error 1 make: Leaving directory `/home/acferen/tcpreplay-trunk/src' make: *** [all] Error 2
You should only get this error if you're trying to build from SVN. The problem is that you do not have GNU Autogen installed on your system. Either install autogen or download one of the source tarballs.
tcpreplay_opts.h:72:3: error: #error option template version mismatches autoopts/options.h header
You're building from svn and the version of Autogen/AutoOpts? installed on your system is different from the version included in the libopts tearoff in the code. The result is that the src/*_opts.[ch] files generated by the system Autogen have a version mismatch with the tearoff.
The solution is to use: ./configure --disable-local-libopts --disable-libopts-install
libnet/libnet-types.h:36:23: error: ../config.h: No such file or directory
Libnet version 1.1.3-RC has a bug in one of the header files. If you get an error during compilation:
/usr/local/include/./libnet/libnet-types.h:36:23: error: ../config.h: No such file or directory
Then you should edit /usr/local/include/libnet/libnet-types.h (or wherever it is installed) and comment out or delete the three lines (should be lines 35-38):
#ifdef HAVE_CONFIG_H #include "../config.h" #endif
Feel free to bitch to libnet@… since they seem to be ignoring me.
Note: Tcpreplay no longer supports libnet!
configure: error: Unable to determine version of libnet
This is usually caused by one of two reasons:
- You have two versions of libnet installed on your system and configure can't figure out which to use. Upgrading to the latest 3.x version and using --with-libnet should fix that.
- You have not actually run 'make install' for libnet and installed it on your system. You can't point --with-libnet to the libnet source tree.
Note: Tcpreplay no longer supports libnet!
Issues with autogen/libopts
The Tcpreplay suite uses the GNU Autogen/libops library. This makes development much easier, but not without some cost. Basically, Autogen/libopts doesn't always provide good backwards compatibility, so if you have a different version of Autogen/libopts then the author does, you may have problems compiling tcpreplay. One example error would be:
make: Entering directory `/root/services_from_source/tcpreplay-3.0.beta11/src/tcpedit' if gcc -DHAVE_CONFIG_H -I. -I. -I../../src -I.. -I../common -I../.. -g -O2 -Wall -O2 -funroll-loops -std=gnu99 -MT tcpedit.o -MD -MP -MF ".deps/tcpedit.Tpo" -c -o tcpedit.o tcpedit.c; \ then mv -f ".deps/tcpedit.Tpo" ".deps/tcpedit.Po"; else rm -f ".deps/tcpedit.Tpo"; exit 1; fi In file included from tcpedit.c:46: tcpedit_stub.h:18:30: autoopts/options.h: No such file or directory In file included from tcpedit.c:46: tcpedit_stub.h:50: error: syntax error before "const" tcpedit_stub.h:50: warning: type defaults to `int' in declaration of `tcpedit_tcpedit_optDesc_p' tcpedit.c:58: error: syntax error before "const" tcpedit.c:58: warning: type defaults to `int' in declaration of `tcpedit_tcpedit_optDesc_p' make: *** [tcpedit.o] Error 1
The good news is that there is an easy fix: ./configure --enable-local-libopts
Problems with linking under recent Fedora Core/Red? Hat
Newer versions of Fedora Core and Red Hat ES/WS do not ship static libraries at all and only have dynamic libraries. If you wish to compile tcpreplay from source, you will need to pass the --enable-dynamic-link to configure in order for tcpreplay to link to them.
Unable to send packet: Error with pcap_inject(packet #10): send: Message too long
There is a bug in OS X (at least up to 10.4.9) which causes problems for ethernet frames > 1500 bytes from being sent when you spoof source MAC addresses (like tcpreplay/tcpbridge does). Apple fixed this in 10.5 (Leopard). Currently there is no work around.
Can't open eth0: libnet_select_device(): Can't find interface eth0
Generally this occurs when the interface (eth0 in this example) is not up or doesn't have an IP address assigned to it.
Can't open lo: libnet_select_device(): Can't find interface lo
Version 1.1.0 of Libnet is unable to send traffic on the loopback device. Upgrade to a later release of the Libnet library to solve this problem.
Libnet can't open hme0: libnet_select_device(): ifaddrlist : SIOCGIFCONF: Invalid argument
Current versions of libnet don't support Solaris network interfaces with many subinterfaces (aka aliased interfaces). Either delete some subinterfaces, use a different operating system or complain to the libnet developers.
Can't open eth0: UID != 0
Tcpreplay requires that you run it as root.
100000 write attempts failed from full buffers and were repeated
When tcpreplay displays a message like "100000 write attempts failed from full buffers and were repeated", this usually means the kernel buffers were full and it had to wait until memory was available. This is quite common when replaying files as fast as possible with the "-R" option. See the tuning OS section in this document for suggestions on solving this problem.
Invalid mac address: 00:00:00:00:00:00
Oldver versions of tcpreplay (prior to 3.x) reserves the MAC address of 00:00:00:00:00:00 as reserved for internal use. Hence you can't rewrite the MAC address of packets to be all zeros.
Unable to process test.cache: cache file version missmatch
Cache files generated by tcpprep and read by tcpreplay are versioned to allow enhancements to the cache file format. Anytime the cache file format changes, the version is incremented. Since this occurs on a very rare basis, this is generally not an issue; however anytime there is a change, it breaks compatibility with previously created cache files. The solution for this problem is to use the same version of tcpreplay and tcpprep to read/write the cache files. Cache file versions match the following versions of tcpprep/tcpreplay:
- Version 1:
Prior to 1.3.beta1
- Version 2: 1.3.beta2 to 1.3.1/1.4.beta1
- Version 3: 1.3.2/1.4.beta2 to 2.0.3
- Version 4: 2.1.0 and above. Note that prior to version 2.3.0, tcpprep had a bug which broke cache file compatibility between big and little endian systems.
Skipping SLL loopback packet.
Your capture file was created on Linux with the 'any' parameter which then captured a packet on the loopback interface. However, tcpreplay doesn't have enough information to actual send the packet, so it skips it. Specifying a destination and source MAC address (-D and -S) will allow tcpreplay to send these packets.
Packet length (8892) is greater then MTU; skipping packet.
The packet length (in this case 8892 bytes) is greater then the maximum transmition unit (MTU) on the outgoing interface. Tcpreplay must skip the packet. Alternatively, you can specify the -T option and tcpreplay will truncate the packet to the MTU size, fix the checksums and send it.
tcpreplay doesn't send entire packet/tcprewrite truncates packets
You won't see any error from tcpreplay, but sometimes you'll open a pcap file in Ethereal/Wireshark? and notice that a packet is something like 400 bytes but tcpreplay says it only sent 100 bytes. I've seen this 2 or 3 times and in each case the reason was the same: the pcap file was broken. Apparently certain older versions of libpcap which shipped with Red Hat Linux had a bug.
First a little background on the pcap file format. At the beginning of each file there is a file header which contains among other things the snaplen. The snaplen is the maximum amount of data that is stored in file per packet; if a packet is larger then this value, the packet is truncated. Each packet is also prefixed with a packet header which contains two values: the len and caplen. The len is the original packet size and the caplen is the amount of actual data which was stored.
A properly formatted pcap file will never have a caplen > snaplen. Some tools and applications unfortunately do not seem to enforce this. So when the libpcap library reads these files, it returns the snaplen as the actual data available. The reason Ethereal/Wireshark? show the entire packet is because they do not use libpcap to read pcap files.
To fix the problem, locate the 2 bytes at offset 16 (0x10) and change them to read 0xFFFF. This will repair the file so libpcap/tcprewrite/tcpreplay/etc can process it correctly.
example of broken file:
xxd broken.pcap | head -3 0000000: d4c3 b2a1 0200 0400 0000 0000 0000 0000 ................ 0000010: 6000 0000 0100 0000 1b6f 954b ca25 0e00 `........o.K.%.. 0000020: 4a00 0000 4a00 0000 0000 5e00 0101 0015 J...J.....^.....
example of fixed file:
xxd fixed.pcap | head -3 0000000: d4c3 b2a1 0200 0400 0000 0000 0000 0000 ................ 0000010: ffff 0000 0100 0000 1b6f 954b ca25 0e00 .........o.K.%.. 0000020: 4a00 0000 4a00 0000 0000 5e00 0101 0015 J...J.....^.....
tcpreplay is sending packets out of order
This isn't a problem with tcpreplay, but rather with the receiving network card/driver. I know for example Broadcom 10G network cards using "multi_mode" prevents tcpdump/Wireshark from seeing the packets in the correct order. The solution is to configure the bnx2x driver with multi_mode=0. Apparently this was turned on by default as of Linux kernel v2.6.24.
This is a list of tickets which people should be aware of since they affect the usage of tcpreplay.
- Selecting only some packets to send (filtering)
- Gracefully handle broken pcaps where snaplen < caplen
- Unable to send full sized packets on OS X
- --seed is fully random on some platforms
- Unable to spoof source MAC address on OS X
- tcpreplay causes OS X WiFi to disassociate from AP
- tcprewrite says "skipping packet" but doesn't
- Document building autogen/guile under Cygwin
- replaying packets over a slip connection in windows fails
- --accurate no longer works
- Compile fails under Sun Studio 11
- BUS ERROR on 64bit SPARC systems
- tcpbridge under OSX & older *BSD is broken
- 802.11 frame injection
- --fixlen=del does not work
- tcprewrite truncates packets
- make fails on cygwin (tcpreplay 3.4.4)
- tcpreplay -p and -M option not worked correctly
- tcpreplay, unable to replay pcap file
- Making a flow from eth0 to eth1 on a virtual machine
- dnet.h should be changed to dumbnet.h for compilation under Backtrack5
- missmatch between dual interface mode & meaning
- --seed results in duplicate IP addresses
- long pcaps cause results to be broken