Performance Testing (Part 2) - Going Commercial

Some time ago I wrote the first installment of a multi-part series on performance testing. Here we go with part two!

In part one I talked about some of the difficulties surrounding performance testing - functionality, flexibility, high scale, quality metrics, etc.

After looking at a couple commercial products we discovered a few things:

1)  Some of these products really do “have it all”.
2)  They can be very expensive.

There was some initial sticker shock ($100,000+) but looking back I shouldn’t have been so surprised.  My first reaction (of course) was to reach out to a few people in the open source community with a proposal.  In a classic “build vs. buy” scenario I wanted to build.  This is (roughly) what I needed:

- At least 40,000 RTP streams (20,000 calls)
- At least 100,000 SIP calls and/or registrations
- The ability to emulate multiple user agents (VLAN, MAC address, DHCP client, SIP UA)
- RTP stream analysis on each RTP leg (MOS, R-Factor, PESQ, etc)
- Flexible device emulation - SIP headers, supported SIP features, etc
- Multiple codec support (at least G711, G729, G722, SILK/OPUS with FEC et
all, etc).
- Control of test scenarios - CPS, number of calls, duration of call,
total duration of test, etc
- Ability to save/load tests via web interface (for ease of use,
comparison of results, etc)
- Ability to perform feature testing - generate DTMF during a call to
navigate an IVR, for example.
- A modular system for monitoring the device under test - Linux load,
CPU usage, disk usage, network I/O, etc.  Could also monitor Cisco
switches in between devices, Windows hosts, etc.  Maybe even
FreeSWITCH or Asterisk if that's what was running on the device under
test.
- Saving and graphing of all relevant performance data - call setup
time, delay, duration, RTP jitter, packet loss, RTP stream stats, etc.
Ability to save data and generate reports from said data.
- Scalable design with master/slave architecture to scale across hosts
with the ability for hardware.

Did I mention this tool needs to be usable by various test engineers, some of which don’t know the difference between SIP and SDP (and rightfully so, they shouldn’t need to)?

With the open source software already available I figured this could be made available for less than the cost of a commercial testing solution.

I gave it away with the title of this post but you can guess what happened next: it was going to cost far more to develop everything I needed.  By the way - it would also take six months to build and take 10 years off my life hunting bugs, managing the project, etc.

BUY

For less than what it would cost to build everything above I could buy this.  A multi-user chassis with one Xcellon-Ultra NP load module and room for another one.  180,000 emulated SIP endpoints.  96,000 RTP streams.  Wire speed 10 gigabit VoIP testing (and 12 gigabit ports).

Of course this isn’t a perfect world...  The chassis runs Windows.  The client software is only available for Windows and the interface is probably the furthest from what I want.  As a guy that eats, lives, and breathes CLI (and has for a decade) multi-pane/dropdown/hidden/shadow GUIs are NOT my thing.  I don’t even know what to call or how to describe some of the window/GUI elements present in the IxLoad user interface...

Ixia even has a solution to this problem.  They offer TCL scripting with clients for various platforms, including Linux!  While we’ll eventually get into that for the time being we went with a much simpler solution:  we setup a Windows terminal server.  I use CoRD from my Mac, login to the terminal server, and run a test.  As you’ll see in part three - IT JUST WORKS.