Network performance measurement

Submitted by sean on Tue, 01/05/2010 - 10:08

Just a few brief notes on a test a while back...

SSH

SSH can be used to also provide performance and delay measurements.
1.    Performance: Create a large file (e.g. 750MB) and then copy it via ‘scp’ between a sever pair, copy the file in both directions to verify symmetry.

dd if=/dev/zero of=/tmp/1 bs=1024 count=768000
scp /tmp/1   server_1:/tmp/1
scp server_1:/tmp/1 /tmp/2

2.    Delay - Visual measurement of delays in an interactive SSH Session:
To “visually” corroborate the TCP delays between the measurement pairs, login from server_2 connect to server_1 (via SSH), and issue the following:

while true; do echo '#############################################################'; done 

To stop press ^C

ICMP/PING

In order to “visually” verify that connectivity is ok before measurements start and to corroborate packets being lost at the IP level, let ping run between the measurements pairs, in each direction.  Use an unloaded pair, not iperf (which will distort due to the system load).

Start ping before failover and stop when complete. Note the number of icmp packets lost.

iperf throughput Test

Iperf is available under Windows and Linux ("apt-get install iperf" on Ubuntu), but works better on linux.

Use one iperf connection connection pair to generate throughput for 60 seconds, with 10 seconds measurement interval on the iperf client.

Server side

$ iperf –s –p 5001

Client side

$iperf –c <ip> -p 5001 –i 10 –t 60

The throughput measurement in MBit/s are shown in sever console and client console (with 10 sec interval). To add more iperf threads use –P flag for iperf client.

Using iperf to generate large loads

The following is a description for a specific test on certain hardware, it may help is the design of your own testsing with iperf.

Configure each client/server pair to generate 100 stable TCP connections. Then launch 50 stable pairs of client/server. This means that we are able to generate up to 5000 TCP connections per server pair. Then install say 10 pairs in paralllel to get 50'000 connections.
This generate a nice load to pump through a switch/router/firewall to test its robustness, and to see how long a failover under load takes.

Use ping and ssh (see above) to measure reponse during the load test.

There is an option in iPerf which allows to generate traffic in both directions (-d), but this gave pretty bad results. Therefore we started 25 client/server pairs in one direction and 25 client/server pairs in the other direction

Notes

  • Gettin iperf to generate max. sessions without hanging the server is not so easy, its depends on the hardware for example. Make sure you have access to the console when doing tests.
  • Scripts were written to automated the above, but they very very specific and cannot be published, sorry.

Commercial tools: ixchariot

There are several commercial tools for load and performance testing. They often cost ten or hundred sof thousands.

One tool that was used was ixchariot, which generate load betweek 10 server pairs. Its quite nice but did not allow creating many parallel sessions (like the 50k iperf sessions above) - which is needed to stress test firewalls.

Tcp/udp packet loss tool

A colleague, Stephane Grundschober, wrote a tool to measure tcp delay and count udp packets lost. Its very useful, and should be published soon on https://sourceforge.net/projects/packetloss/.

 Further reading

http://sourceforge.net/projects/iperf/

Other tools found but not yet analysed:

http://www.cs.unc.edu/~jasleen/Research-passivetcp.htm
http://www.hep.man.ac.uk/u/rich/Tools_Software/tcpdelay.html
tcpperf   http://www.wand.net.nz/~stj2/nsc/software.html